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ABSTRACT 


The  purpose  of  this  thesis  is  to  study  the  usability  an  upgraded  Maintenance  Error 
Information  Management  System  (MEIMS)  tool  used  to  capture  human  error  in  Naval 
Aviation  maintenance  mishaps.  Built  upon  the  Human  Factors  Analysis  and 
Classification  System-Maintenance  Extension  taxonomy,  the  tool  provides  the 
framework  for  examining  maintenance  errors  that  lead  to  mishaps,  incidents;  and  ■ 
personal  injuries.  The  tool  is  developed  for  safety  personnel,  mishap  investigators. 
Aircraft  Mishap  Board  members,  and  analysts.  In  limited  usability  testing,  the  tool  was 
found  to  be  useful,  but  in  need  of  revisions,  specifically  regarding  functionality  and  user 
friendliness.  Additionally,  a  tutorial  is  provided  to  better  prepare  targeted  users  of  the 
tool.  The  study  requires  a  review  of  mishap  information  systems,  human  error  theories 
related  to  aviation  mishaps,  design  considerations  for  human-computer  interfaces  and 
usability  study  applications.  A  follow-on  usability  study,  conducted  using  two  groups  of 
potential  users,  one  which  received  the  tutorial  and  one  which  did  not.  It  includes  a 
survey  regarding  subjective  responses  about  the  prototype  tool.  The  results  indicate  that 
the  tutorial  is  effective  in  preparing  and  assisting  potential  users,  and  that  the  tool  could 
make  a  significant  impact  in  the  reduction  of  mishap  rates  due  to  maintenance  error. 
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I.  INTRODUCTION 


A.  OVERVIEW 

1.  Background 

In  the  last  half  century.  Naval  Aviation  has  reduced  the  Class  A  Flight  Mishaps 
(FM)  rate  significantly.  In  fact,  the  Class  A  FM  rate  was  reduced  by  50  percent  in  each 
decade  from  1950  to  2000  (see  Figure  1).  Despite  this  success,  the  cost  of  mishaps,  in 
terms  of  lives  and  assets,  remains  too  high.  Furthermore,  human  error,  attributable  to 
four  out  of  every  five  mishaps,  has  remained  constant  (Nutwell  &  Sherman,  1997).  In 
response  to  a  1996  F-14  Tomcat  FM,  which  was  largely  attributable  to  human  error,  a 
Human  Factors  Quality  Management  Board  (HFQMB)  was  initiated  to  study  human  error 
in  Naval  Aviation  mishaps.  The  HFQMB ’s  charter  (1997)  was  to  reduce  the  mishap  rate 
by  identifying  systematic  improvements  in  processes  and  systems  that  affect  human 
performance  and  to  institutionalize  continual  improvements  in  these  areas.  The  HFQMB 
is  made  up  of  Flag  Officers,  senior  operational  officers,  and  a  cross  section  of 
Commanding  Officers  to  seasoned  naval  aviators  from  all  communities.  Their  objective 
was  to  reduce  the  Class  A  FM  rate  by  half  within  three  years,  and  by  75  percent  within  10 
years  (HFQMB,  1997). 

The  HFQMB ’s  strategy  entailed  a  three-pronged  approach:  (1)  Mishap  Data 
Analysis  (MDA),  (2)  Organizational  Benchmarking  (OB),  and  (3)  Command  Safety 
Assessment  (CSA).  MDA  studies  historical  records  to  determine  established  trends  of 
human  error  in  aviation  mishaps.  The  Human  Factors  Analysis  and  Classification 
System  (HFACS)  was  developed  to  accomplish  this  task.  By  examining  and  classifying 
past  mishaps,  prevailing  human  errors  are  prioritized  and  targeted  for  prevention.  The 
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second  phase,  OB,  involved  the  observance  of  other  organizations,  both  military  and 
civilian,  for  procedures  and  practices  that  might  be  useful  in  helping  reduce  the 
occurrence  of  human  error.  To  this  end,  Naval  Aviation  adopted  Organizational  Risk 
Management,  which  associates  conditions  with  risk  and  decision  making  methodology  to 
reduce  the  occurrence  of  aviation  mishaps,  directly  from  the  U.  S.  Army  (Department  of 
the  Navy,  1997).  CSA  was  developed  at  the  Naval  Postgraduate  School  by  Civarelli  and 
Figlock  (1997)  as  a  tool  to  measure  the  safety  climate  in  a  squadron  from  an  aircrew 
perspective.  It  solicits  opinions  and  attitudes  towards  procedures  and  practices  to 
determine  the  effect  of  organizational  and  supervisory  issues  on  flight  safety  (Nutwell  & 
Sherman,  1997). 


Fiscal  Year  (FY) 


Figure  1:  FY  1950-1999  Naval  Aviation  Class  A  Flight  Mishap  Rates 
(From  School  of  Aviation  Safety,  1999) 

In  January  1999,  the  Vice  Chief  of  Naval  Operations  extended  the  short-term  goal 
of  50  percent  reduction  in  Class  A  FM  mishap  rate  from  the  end  of  FY 1999  to  the  end  of 
FY2000.  To  meet  this  goal,  the  scope  of  the  HFQMB  needed  to  expand  to  include 
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human  error  in  maintenance  mishaps.  Historically,  approximately  one  in  every  five  Class 
A  FMs  contains  maintenance  error.  By  expanding  the  focus  to  include  maintenance 
error,  it  was  believed  that  the  50  percent  reduction  objective  could  be  met.  An  identical 
three-prong  approach  that  had  been  used  for  aircrew  error  was  utilized  for  maintenance 
error. 

2.  Maintenance  Mishap  Data  Analysis 

HFACS  was  developed  as  a  framework  for  examining  aircrew  and  supervisory 
error  in  Class  A  FMs.  It  concurs  with  the  Naval  Aviation  Safety  Program’s 
(OPNVAVINST  3750.6)  notion  of  multiple  causal  factors,  chain  of  events,  and  human 
factors.  A  maintenance  extension  (ME)  was  added  to  HFACS  to  meet  the  HFQMB’s 
expanded  scope  and  was  used  to  examine  incidents  and  injuries  (Schmidt,  Figlock  & 
Teeters,  1999),  and  major  to  minor  mishap  data  (Schmidt,  Schmorrow  &  Figlock,  2000). 
HFACS-ME  was  adopted  for  the  upcoming  revision  of  the  Naval  Aviation  Safety 
Program.  Furthermore,  a  laptop  prototype  tool.  Maintenance  Extension  Information 
Management  System  (MEIMS)  was  developed  to  allow  users  to  collect,  catalog,  collate 
and  analyze  mishap  data,  to  better  identify  trends  and  for  safety  training.  MEIMS,  in 
limited  usability  testing,  was  determined  to  be  a  valuable  tool  that  will  provide  the  fleet 
great  benefit  (Wood,  2000). 

B.  PROBLEM  STATEMENT 

To  meet  the  objective  of  a  50  percent  reduction  in  Class  A  FMs,  both  aircrew  and 
maintenance  error  needed  to  be  targeted.  The  HFACS-ME  taxonomy  and  MEIMS  were 
developed  to  better  analyze  maintenance  error.  While  proven  useful  in  a  limited  usability 
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study,  MEEMS  is  not  without  limitations.  Users  found  the  tool  to  be  somewhat  difficult 
to  navigate,  was  lacking  in  assistance  functionality  and  its  interface  needed  to  be  more 
user-friendly.  Additionally,  it  was  determined  that  its  intended  users  (e.g.,  squadron 
maintenance  and  safety  personnel,  squadron  safety  officers,  Aircraft  Mishap  Board 
members.  Naval  Safety  Center  analysts,  etc)  would  benefit  from  a  training  tutorial  prior 
to  accessing  the  tool  (Wood,  2000).  This  study  will  further  develop  MEIMS  based  upon 
the  initial  usability  study.  It  is  believed  that  the  fleet  user,  with  proper  training,  should  be 
able  to  easily  access  the  tool  and  obtain  useful  information,  which  can  then  be  used  in 
training,  hazard  identification  and  trend  analysis,  to  prevent  future  mishaps. 

This  thesis  investigates  the  following  questions: 

1 .  What  query  capabilities  are  required  by  the  users  of  the  information 
system  to  best  identify  problem  areas  and  trends? 

2.  What  additions  will  make  the  information  system  more  user-friendly, 
allowing  efficient  and  effective  access  to  the  system’s  capabilities? 

3.  What  kind  of  training,  in  the  form  of  a  tutorial,  would  best  prepare  users  to 
efficiently  and  effectively  access  the  information  system’s  full 
capabilities? 

C.  PURPOSE 

The  intent  of  this  study  is  to  refine  and  evaluate  the  existing  safety  information 
management  system,  which  uses  the  HFACS-ME  taxonomy  as  its  basis,  to  better 
facilitate  data  collection,  organization,  query,  analysis  and  reporting  of  maintenance 
personnel  errors  that  contribute  to  Naval  Aviation  mishaps,  equipment  damage  and 


4 


personnel  injury.  Included  are  theoretical  approaches  that  examine  mishaps  involving 
human  error:  Heinrich’s  “Domino”  Theory,  Edwards’  “SHEL”  Model,  and  Reason’s 
“Swiss  Cheese”  Model.  These  models  not  only  help  identify  the  unsafe  actions  which 
cause  mishaps,  but  they  also  connect  the  conditions  underlying  the  mishap.  HFACS-ME 
is  a  composite  derivative  of  these  taxonomies.  Although  there  is  no  universally  accepted 
method  of  accident  investigation  (Benner,  1975),  some  standardized  aircraft  accident 
investigation  procedures  have  been  adopted  by  most  agencies  throughout  the  world 
(Diehl,  1991).  Using  HFACS-ME  as  its  basis,  MEIMS  will  significantly  improve  access 
and  analysis  of  mishap  data. 

The  results  of  this  study  will  (1)  improve  and  enhance  the  MEIMS  prototype 
tool’s  usability  so  that  data  can  be  easily  queried  and  analyzed,  (2)  creation  of  a  tutorial 
for  fleet  customers  that  will  enhance  their  ability  to  not  only  access  the  data  base,  but  to 
efficiently  and  effectively  use  this  information  to  prevent  future  mishaps,  and  (3)  study 
the  performance  of  two  groups  of  participants,  with  only  one  group  receiving  the  tutorial, 
to  determine  the  effectiveness  of  the  tutorial  for  fleet  users. 

D.  SCOPE  AND  LIMITATIONS 

MEIMS  will  be  revised  to  include  a  users’  tutorial.  Fleet  personnel,  primarily 
Aviation  Safety  Officers,  will  test  MEIMS  to  determine  its  usability  and  the  benefits  of 
the  help  and  tutorial  functionality.  The  fully  developed  MEIMS  tool  will  be  used 
primarily  by  Naval  Aviation  squadrons,  but  may  have  some  crossover  use  by  other 
military  branches  and  civilian  airlines.  Only  maintenance  related  mishaps  caused  by 
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human  error  are  considered.  No  material  failure  factors  or  maintenance  related  hazard 
reports  or  personnel  injuries  not  related  to  a  mishap  are  included. 

E.  DEFINITIONS 

This  study  uses  the  following  definitions: 

Aircraft  Mishap  Board:  Group  of  officers  appointed  to  investigate  and  report  on 
an  aviation  mishap. 

Aviation  mishap  rate:  Number  of  aviation  mishaps  per  100,000  flight  hours. 
Aviation  Safety  Officer:  Principal  advisor  to  Naval  Aviation  squadron 
commanding  officers  on  all  aviation  safety  matters. 

F-14  Tomcat:  U.  S.  Navy’s  swing-wing,  supersonic  fighter  aircraft  with  air-to-air, 
air-to-ground,  and  reconnaissance  capability. 

Fleet  Logistics  Support  Wing:  U.  S.  Navy  reserve  air  wing  comprised  of  transport 
aircraft. 

HFACS:  Human  Factors  Analysis  and  Classification  System  designed  to  help 
analyze  Naval  Aviation  mishaps  focusing  on  aircrew  error. 

HFACS-ME:  Human  Factors  Analysis  and  Classification  System-Maintenance 
Extension  adaptation  to  classify  causal  factors  that  contribute  to  maintenance  mishaps. 

HFOMB:  Human  Factors  Quality  Management  Board  established  by  Naval 
Aviation  senior  leadership  to  reduce  human  error  involvement  in  Naval  Aviation  Class  A 
flight  mishaps. 

MEIMS:  Maintenance  Error  Information  Management  System,  a  prototype  tool 
developed  to  collect,  catalog,  collate  and  analyze  mishap  data. 
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Mishap:  A  naval  mishap  is  an  unplanned  event  or  series  of  events  directly 
involving  naval  aircraft,  which  result  in  $10,000  or  greater  cumulative  damage  to  naval 
aircraft,  other  aircraft,  property,  or  personnel  injury. 

Mishap  Categories:  Naval  aircraft  mishap  categories  are  defined  below: 

Flight  Mishap  (FM):  Those  mishaps  in  which  there  was  $10,000  or 
greater  DOD  aircraft  damage  or  loss  of  a  DOD  aircraft,  and  intent  for  flight  for 
DOD  aircraft  existed  at  the  time  of  the  mishap.  Other  property  damage,  injury,  or 
death  may  or  may  not  have  occurred. 

Flight  Related  Mishap  (TRM):  Those  mishaps  in  which  there  was  less 
than  $10,000  DOD  aircraft  damage,  and  intent  for  flight  (for  DOD  aircraft) 
existed  at  the  time  of  the  mishap,  and  $10,000  or  more  total  damage  or  a  defined 
injury  or  death  occurred. 

Aircraft  Ground  Mishap  (AGM):  Those  mishaps  in  which  no  intent  for 
flight  existed  at  the  time  of  the  mishap  and  DOD  aircraft  loss,  or  $10,000  or  more 
aircraft  damage,  and/or  property  damage,  or  a  defined  injury  or  death  occurred. 
Mishap  Severity  Class:  Mishap  severity  based  on  injury  and  property  damage. 

Class  A:  A  mishap  in  which  the  total  cost  of  property  damage 
(including  all  aircraft  damage)  is  $1,000,000  or  greater;  or  a  naval  aircraft  is 
destroyed  or  missing;  or  any  fatality  or  permanent  total  disability  occurs  with 
direct  involvement  of  naval  aircraft. 

Class  B:  A  mishap  in  which  the  total  cost  of  property  damage 
(including  all  aircraft  damage)  is  $200,000  or  more,  but  less  than  $1,000,000 
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and/or  a  permanent  partial  disability,  and/or  the  hospitalization  of  five  or  more 
personnel. 

Class  C:  A  mishap  in  which  the  total  cost  of  property  damage 
(including  all  aircraft  damage)  is  $10,000  or  more  but  less  then  $200,000  and/or 
injury  results  in  five  or  more  lost  workdays. 

Naval  Aircraft:  Refers  to  U.S.  Navy,  Naval  Reserve,  and  Marine  Corps  aircraft. 
OPNAVINST  3750.6:  The  Naval  Aviation  Safety  Program:  U.S.  Navy  instruction 
outlining  Naval  Aviation’s  safety  program. 

QRM:  Operational  Risk  Management.  A  decision  making  tool  to  increase 
effectiveness  by  anticipating  and  reducing  hazards,  thus  increasing  the  probability  of  a 
successful  mission. 

F.  ORGANIZATION  OF  STUDY 

Chapter  II  contains  a  literature  review  on  the  development  of  a  prototype  to 
identify  human  error  involvement  and  patterns  in  aviation  maintenance  mishaps.  The 
methods  used  in  this  study  are  discussed  in  Chapter  III.  The  results  of  this  study  are 
presented  in  Chapter  IV.  Finally,  Chapter  V  contains  conclusions,  findings,  and 
recommendations . 
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II.  LITERATURE  REVIEW 


A.  OVERVIEW 

The  literature  studied  encompasses  human  error,  maintenance  error  in  aviation, 
error  classification  and  analysis.  It  includes  textbooks,  research  articles,  and  masters 
thesis  pertaining  to:  (1)  human  error  theories,  and  their  relation  to  maintenance  related 
aviation  mishaps,  (2)  accident  information  management,  and  (3)  design  and  usability 
assessment  of  software  tools.  This  information  provides  the  basis  for  the  continuing 
development  of  the  maintenance  error  analysis  and  reporting  prototype  tool. 

While  ongoing  efforts  to  reduce  the  number  of  Class  A  mishaps  in  Naval  Aviation  are  to 
be  applauded,  there  is  potential  for  further  improvement,  especially  in  the  area  of  human 
error.  When  examining  mishaps  involving  human  error  there  are  numerous  theoretical 
approaches  from  which  to  choose  (Goetsch,  1996).  Some  are  founded  in  industrial 
safety;  others  are  viewed  from  a  complex  systems  perspective,  emphasizing  human 
factors  and  operator  error.  Some  approaches  use  models  drawn  from  the  domain  of 
preventive  medicine  that  employ  epidemiological  factors  to  analyze  accidents,  while 
other  models  utilize  a  combination  of  these  approaches  (Schmidt,  1998). 

B.  ACCIDENT  DATA  AND  MANAGEMENT 

1.  Investigation  and  Reporting 

Diehl’s  (1991)  three-stage  model  of  accident  investigation  and  prevention  focuses 
on  human  performance  and  systems  safety  considerations  (see  Figure  2).  Diehl’s  stages 
are  Accident  Generation,  Accident  Investigation  Process,  and  Preventive  Measures.  The 
purpose  of  the  first  stage,  Accident  Generation,  is  the  identification  of  hazards.  Hazards 
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are  circumstances  or  conditions  which  have  the  potential  to  lead  to  an  incident  (near¬ 
accident)  or  even  an  accident.  The  second  stage,  Accident  Investigation  Process, 
includes  the  collection,  analysis,  and  review  of  accident  data  and  the  focus  of  this  review. 
In  aircraft  accident  investigations,  procedure  calls  for  a  fact  finding  body  to  investigate 
the  accident  to  determine  what  happened,  and  subsequently  what  caused  accidents. 
Lastly,  a  final  report  is  made,  which  details  causes  of  accidents  and  makes 
recommendations  to  prevent  reoccurrence  (Diehl,  1991). 


Figure  2:  Accident  Generation,  Investigation,  and  Prevention  Elements 

(After  Diehl,  1989) 

The  final  stage.  Prevention  Measures,  details  the  methods  used  to  avoid  future 
accidents.  There  are  four  categories  of  accident-prevention  measures:  (1)  eliminating 
hazards  and  risks,  (2)  incorporating  safety  features  (3)  providing  warning  devices,  and 
(4)  establishing  procedural  safeguards.  In  ascending  order,  these  measures  are  more 
effective,  but  are  also  more  expensive  and  more  difficult  to  attain  (Diehl,  1991). 
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2.  Accident  Prevention 


In  the  early  1900s,  accident  investigations  were  based  on  the  notion  people 
commit  unsafe  acts  that  lead  to  incidents  (Heinrich,  1959).  This  blame-based  practice 
deterred  systematic  accident  prevention  until  the  1950’s,  when  systems  engineering 
emerged  from  of  the  U.S.  military’s  large-scale  weapons  program.  Applied  to  accident 
prevention,  systems  engineering  focuses  on  the  strengths  and  limitations  of  components, 
including  the  human  element,  as  an  integral  part  of  the  system  (Heinrich,  Petersen  & 
Roos,  1980).  Presently,  up  to  90  percent  of  accidents  can  be  attributable  to  human  error 
in  a  variety  of  government,  military  and  industrial  settings  (Heinrich,  Petersen,  &  Roos, 
1980;  Hale  &  Glendon,  1987;  School  of  Aviation  Safety,  2000),  so  analyzing  the  human 
element  can  vastly  improve  the  understanding  of  systems  and  why  they  fail. 

Operational  Risk  Management  (ORM),  an  outgrowth  of  systems  engineering,  is  a 
tool  that  the  armed  forces  employed  to  decrease  accidents.  ORM  focuses  on  the 
identification  of  hazards  during  mission  planning  to  effect  control  measures  which  reduce 
the  associated  risk.  It  is  especially  effective  in  highlighting  human  factor  hazards  (DON, 
1997).  The  U.S.  Army’s  aviation  branch  achieved  record  low  accident  rates  in  1995  and 
1996,  largely  attributable  to  their  employment  of  ORM  (DO A,  2000).  Since  1997,  the 
other  military  services  institutionalized  ORM  into  their  operational  doctrine  (School  of 
Avaition  Safety,  2000). 

3.  Data  Management 

To  be  useful  in  the  prevention  of  accidents,  relevant  data  must  be  collected  and 
properly  archived  for  future  reference.  Coding  the  data  and  the  use  of  data  bases  have 
become  universally  accepted  methods  for  this  task.  The  National  Safety  Council 
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established  a  numerical  code  system  relating  to  each  cause  factor  (National  Safety  News, 
1975).  This  method  of  coding  simultaneously  made  referencing  for  specific  cause  factors 
much  simpler,  and  also  provided  more  efficient  analysis  capability  by  focusing  on 
specific  causes  rather  than  accidents  as  a  whole.  Additionally,  it  is  essential  to  also 
include  “near-miss”  incident  data,  as  their  circumstances  are  as  likely  to  cause  an 
accident  in  the  future.  Computer  analysis  tools  can  significantly  aid  reviews  of  mishap 
histories  (Grimaldi  &  Simonds,  1984).  To  be  truly  effective  in  reducing  future  accidents 
and  incidents  those  tools  must  organize  and  tabulate  the  data  logically,  and  present  it  for 
analysis  in  easily  accessible,  user  friendly  formats. 

C.  ACCIDENT  CAUSATION  THEORIES 

There  are  numerous  theoretical  approaches  to  examine  mishaps  involving  human 
error  (Goetsch,  1996).  Three  of  the  most  recognized  approaches  are  Heinrich’s  Domino 
Theory,  Edwards’  SHEL  Model  and  Reason’s  Swiss  Cheese  Model.  Despite  differences 
in  approach  and  originating  source,  these  theories  compliment  each  other  and  provide  a 
solid  foundation  for  analyzing  human  error  in  avaition  mishaps. 

1.  Heinrich’s  “Domino”  Theory 

Considered  to  be  the  original  accident  theory  (Goetsch,  1996),  it  details  a  linear 
five  step  sequence  of  related  factors  (chain  of  events)  that  lead  to  an  actual  mishap.  Bird 
(1980)  also  held  this  view.  The  two  central  principles  of  the  Domino  Theory  are  (Bird, 
1980):  (1)  accidents  are  caused  by  the  combined  actions  of  the  preceding  factors,  and  (2) 
removal  of  the  middle  factor  (unsafe  act  or  condition)  will  negate  the  actions  of  the 
preceding  factors  and  thus  prevent  accidents  and  injuries  (see  Figure  3). 
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Figure  3:  The  Five-Step  Domino  Sequence  (Bird,  1980) 

Heinrich’s  original  five-factor  model  (Heinrich,  Petersen,  &  Roos,  1980),  can  be 
paraphrased  as  follows: 

1 .  Lack  of  Control:  This  is  a  management  issue  where  the  emphasis  is  placed 
on  the  control  exercised  in  a  situation  for  an  array  of  factors. 

2.  Basic  Cause(s):  This  identifies  the  origin(s)  of  the  causes  and  includes 
aspects  such  as  human  factors,  environmental  factors,  or  job-related 
factors. 

3.  Immediate  Cause(s):  This  includes  substandard  practices  and  conditions 
that  are  symptoms  of  the  basic  causes. 

4.  Incident:  This  typically  involves  contact  with  the  hazard,  and  for  example, 
results  in  a  fall  or  impact  with  moving  objects. 

5.  Personal  Injury  and  Property  Damage:  This  includes  lacerations,  fractures, 
death,  and  material  damage. 

Each  step  preempts  the  next,  causing  it  to  occur;  much  the  same  way  as  one 
domino  falling  causes  the  next  domino  in  sequence  to  fall  as  well.  Removal  of  the 
factors  that  comprise  any  of  the  first  three  “dominos”  will  effectively  intervene  to  prevent 
the  accident. 
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2.  Edwards’  “SHEL”  Model 


The  “SHEL  Model”  (see  Figure  4)  of  system  design  was  developed  in  the  early 
1970’s  to  provide  a  better  way  to  evaluate  failures  in  human-machine  systems  (Hawkins, 
1992).  It  identifies  and  defines  four  system  dimensions:  Software,  Hardware, 
Environment,  and  Liveware.  The  “SHEL  Model”  provides  a  method  to  describe  systems, 
identify  potential  areas  for  concern  within  a  system,  and  proposes  a  general  framework 
for  accident  investigation.  Edwards  (1988)  defines  SHEL  concepts  as  follows: 

Software:  the  rules,  regulations,  laws,  orders,  standard  operating  procedures,  customs, 
practices,  and  habits  that  govern  the  manner  in  which  the  system  operates  and  in  which 
the  information  within  it  is  organized.  Software  is  typically  a  collection  of  documents. 

1.  Hardware:  the  buildings,  vehicles,  equipment,  and  materials  of  which  the 
system  is  comprised. 

2.  Environmental  conditions:  the  physical,  economic,  political  and  social 
factors  within  which  the  software,  hardware,  and  liveware  operate. 

3.  Liveware:  the  human  beings  involved  with  the  system. 

According  to  the  “SHEL  Model,”  the  system  will  fail  when  a  disconnect  occurs  in 
any  one  of  the  four  dimensions,  or  in  the  connections  between  them.  People  are  rarely 
the  sole  cause  of  accidents  (Edwards,  1988),  but  rather,  accidents  are  caused  by  the 
interaction  of  several  factors  (Shappell  &  Wiegmann,  1997).  This  multi-factor  theory 
was  a  significant  departure  from  the  commonly  held  belief  that  accidents  were  caused  by 
single  events. 
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3.  Reason’s  “Swiss  Cheese”  Model 

Reason’s  “Swiss  Cheese”  Model  employs  a  vertical  association  human  factors 
approach  to  a  collection  of  factors  that  eventually  lead  to  an  accident.  It  differentiates 
between  two  types  of  errors:  1)  active  failures,  whose  effects  are  felt  immediately,  and 
2)  latent  conditions,  whose  effects  may  lie  dormant  until  triggered  later,  usually  by  other 
mitigating  factors  (Reason,  1990).  Latent  conditions  “set  the  stage”  for  an  accident  while 
active  failures  tend  to  be  the  catalyst  for  the  accident  to  finally  occur.  The  model  can  be 
thought  of  as  slices  of  Swiss  cheese  lined  up,  with  each  vertical  slice  representing  a 
defense  layer  (e.g.  training,  good  management,  teamwork,  etc.)  and  each  hole 
representing  an  active  failure  or  latent  condition  in  that  defense  (see  Figure  5).  Should  a 
situation  where  holes  line  up  come  to  pass,  an  accident  will  occur. 

This  is  a  dynamic  model  with  each  defensive  layer  coming  in  and  out  of 
prominence  according  to  situational  characteristics  (Reason,  1990).  An  event  may  occur 
in  one  of  three  levels:  (1)  person-unsafe  acts,  (2)  workplace-error  provoking  conditions, 
and  (3)  organization-error  establishing  conditions.  Organizational  factors,  in  which 
strategic  decisions  and  associated  processes  (budgeting,  forecasting,  resource  allocation) 
are  initiated,  provide  the  starting  point  for  an  accident  (Reason,  1997).  These  processes, 
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influenced  by  corporate  culture,  are  distributed  throughout  the  organization  to  individual 
workplaces.  Corporate  processes,  evidenced  as  inadequate  staffing,  time  pressures, 
equipment,  training  and  working  conditions,  combine  with  the  natural  proclivity  to 
commit  errors  and/or  violations,  to  result  in  unsafe  acts.  Few  of  these  acts,  however, 
actually  create  holes  in  the  defense  layers  en  route  to  becoming  an  accident. 


Figure  5:  Reason’s  Swiss  Cheese  Model  (Naval  Safety  Center,  1996) 

D.  HUMAN  FACTORS  ANALYSIS  &  CLASSIFICATION  SYSTEM  (HFACS) 

Efforts  to  analyze  aircrew  error  in  Naval  Aviation  mishaps  led  to  the  development 
of  HFACS  (Shappell  &  Wiegmann,  1997).  It  began  as  a  restructuring  and  expansion  of 
the  Swiss  Cheese  Model,  but  it  also  includes  features  of  Heinrich’s  Domino  Theory  and 
Edwards’  SHEL  Model.  The  HFACS  taxonomy  recognizes  the  set  of  conditions  within 
which  system  operators  perform,  accounts  for  interrelationships  among  conditions,  and 
provides  an  overall  sense  of  order  or  implied  hierarchy.  It  identifies  both  active  failures 
and  latent  conditions  within  four  categories  (DON,  2000):  (1)  unsafe  acts;  (2)  pre- 
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conditions  for  unsafe  acts;  (3)  unsafe  supervision,  and  (4)  organizational  influences.  This 
classification  is  then  used  to  target  the  most  appropriate  intervention. 

While  Reason’s  “Swiss  Cheese”  model  is  widely  held  as  integral  to  explaining 
accidents,  it  does  not  provide  a  means  of  delineating  precursors  to  accidents  (Shappell  &, 
Weigmann  1997).  Although  modem  analysis  systems  are  heading  in  the  right  direction 
regarding  human  factors  investigation,  more  effort  is  needed.  There  exists  a  requirement 
for  a  framework  that  allows  for  the  explanation  of  accident  causation  involving  human 
error  to  include  not  only  the  causes  of  those  errors,  but  also  an  explanation  of  those 
causes.  It  is  through  such  efforts  that  further  occurrences  of  the  same  type  of  accident 
can  be  avoided. 

1.  Maintenance  Mishaps 

The  aviation  industry  has,  in  recent  years,  begun  to  apply  human  error  theory  to 
maintenance-related  mishaps  with  the  intent  of  modernizing  accident  investigations 
(Marx,  1998).  Previously,  it  was  believed  that  every  error  could  be  traced  back  to  a  basic 
set  of  actions  and  associated  conditions  that  precipitated  the  error  (Goetsch,  1996).  This 
simplistic  view  did  not  allow  for  errors  to  have  multiple  causes,  as  most  do. 

Additionally,  traditional  investigation  techniques,  while  appropriate  for  identifying  the 
causes  of  equipment  failures,  did  not  have  the  same  success  with  human  error-related 
accidents  (O'Connor  &  Bacchi,  1997).  Typically,  investigations  would  effectively  end 
when  the  cause  pointed  to  human  error,  with  no  effort  expended  in  attempting  to  explain 
why  the  error  occurred. 

In  a  review  of  investigation  and  analysis  systems  for  aircraft  maintenance  error, 
the  need  for  human  factors  investigation  and  reporting  was  identified  (Marx,  1998). 
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Although  human  factors  investigation  methods  are  acknowledged  as  critical  to 
understanding  why  people  make  certain  mistakes,  they  have  not  been  widely  adopted. 
The  proper  investigation  of  human  factors,  however,  is  vital  to  accident  prevention 
(Harle,  1994).  The  lack  of  thorough  human  factors  investigation  was  caused  by  the 
tendency  to  place  blame,  inability  to  see  through  proximate  causes  to  underlying  causes, 
and  an  over-emphasis  on  static  factors  such  as  who,  what  and  when  (Marx,  1998). 

2.  HFACS-ME  Maintenance  Extension 

In  light  of  the  above  requirements,  the  HFACS  taxonomy  was  adapted,  and 
augmented  to  classify  factors  that  lead  to  maintenance  mishaps  (Schmidt,  1996).  The 
Maintenance  Extension  (ME)  of  the  original  HFACS  taxonomy  focuses  on  causation 
factors  particular  to  the  maintenance  environment.  These  aspects  and  their  relevant  sub¬ 
section  classifications  are  given  in  a  manner  similar  to  Reason’s  model.  In  the  HFACS- 
ME  model,  a  maintainer’s  performance  is  seen  as  being  influenced  by  a  series  of  latent 
conditions  (supervisory,  maintainer  and  working  conditions)  that  can  lead  to  an  Unsafe 
Maintainer  Act,  which  in  turn  can  lead  to  a  mishap,  injury  or  an  unsafe  maintenance 
condition  (see  Figure  6)' 
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Figure  6:  HFACS  Maintenance  Extension  (HFACS-ME) 

(DON,  2000) 


HFACS-ME  consists  of  four  broad  human  error  categories,  three  latent: 
Management  Conditions  (e.g.  inadequate  supervision),  Maintainer  Conditions  (e.g. 
preparation/training).  Working  Conditions  (e.g.  lighting/light),  and  the  fourth  Maintainer 
Acts  (e.g.  skill  error),  is  active  (DON,  2000).  The  three  orders  of  maintenance  error 
(first,  second,  and  third)  reflect  a  decomposition  of  the  error  types  from  a  macro  to  a 
micro  perspective.  Each  successive  order  provides  for  greater  granularity,  serving  the 
respective  purposes  of  identifying  problem  areas,  prioritizing  potential  targets,  and 
tailoring  intervention  strategies  (see  Table  1). 
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Table  1:  HFACS  Maintenance  Extension  Categories  (DON,  2000) 


First  Order 

Second  Order 

Third  Order 

Organizational 

Inadequate  Processes 

Inadequate  Documentation 

Inadequate  Design 

Management  Conditions 

Inadequate  Resources 

Supervisory 

Inadequate  Supervision 

Inappropriate  Operations 

Uncorrected  Problem 

Supervisory  Misconduct 

Medical 

Adverse  Mental  State 

Adverse  Physical  State 

Unsafe  Limitation 

Maintainer  Conditions 

Crew  Coordination 

Inadequate  Communication 

Inadequate  Assertiveness 

Inadequate  Adaptability/Flexibility 

Readiness 

Inadequate  Training/Preparation 

Inadequate  Certification/Qualification 
Personnel  Readiness  Infringement 

Environment 

Inadequate  Lighting/Light 

Unsafe  Weather/Exposure 

Unsafe  Environmental  Hazards 

Working  Conditions 

Equipment 

Damaged/U  n  serviced 
Unavailable/Inappropriate 
Dated/Uncertified 

Workspace 

Confining 

Obstructed 

Inaccessible 

Maintainer  Acts 

Error 

Attention/Memory 

Knowledge/Rule 

Skill/Technique 

Judgment/Decision 

Violation 

Routine 

Infraction 

Exceptional 

Flagrant 

3.  Maintenance  Error  Information  Management  System  (MEIMS) 

The  HFACS-ME  taxonomy  was  used  as  the  framework  for  the  development  of  a 
laptop  prototype  database  tool,  MEIMS.  It  is  intended  for  fleet  users  to  collect,  catalog, 
collate  and  analyze  mishap  data,  and  to  better  identify  trends  and  for  safety  training.  In 
limited  usability  testing,  MEIMS  was  proven  to  be  effective,  yet  was  lacking  in  some 
areas.  Specifically  MEIMS  was  somewhat  difficult  to  navigate,  was  lacking  in  assistance 
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functionality  and  it’s  customer  interface  needed  to  be  more  user-friendly.  The  benefit  of 
user  training  was  also  indicated.  To  fully  reach  its  potential,  however,  MEIMS  needs 
design  refinements  and  more  rigorous  usability  testing  (Wood,  2000). 

E.  USER INTERFACE 

The  proliferation  of  computer  systems  into  ever  increasing  environments 
highlights  the  need  for  effective  and  easy  to  use  interfaces.  Clearly,  the  user  interface  is 
the  most  important  factor  in  determining  the  success  or  failure  of  a  software  application 
(Liu,  1997).  The  design  objectives  of  user  interface  include  effectiveness,  efficiency, 
comfort  and  safety.  The  content  of  an  interface  should  abstract  critical  features  of  the 
target  system  (Norman,  1983;  Vicente  &  Rasmussen,  1992),  be  consistent  with  human 
conceptual  and  cognitive  characteristics  (Carroll,  Mack  &  Kellog,  1988;  Wickens,  1992), 
be  compatible  with  human  response  tendencies  (Myers,  1991;  Shneiderman,  1983),  and 
should  help  users  effectively  achieve  their  intended  impact  on  the  target  system 
(Sheridan,  1984;  Shneiderman,  1992). 

Williges,  Williges  &  Elkerton  (1987)  examined  design  objectives  from  user's 
perspective  and  generated  a  list  of  seven  general  design  principles  as  a  basis  for 
developing  specific  design  objectives.  The  principles  are  (1)  consistency,  (2) 
compatibility,  (3)  memory,  (4)  structure,  (5)  feedback,  (6)  workload,  and  (7) 
individualization.  According  to  these  principles,  a  well  designed  user  interface  should 
maintain  the  same  style  of  interaction  within  itself  and  similar  applications  (consistency), 
provide  easily  understandable  interface  symbols  and  icons  to  minimize  the  demand  for 
information  recoding  (compatibility),  minimize  the  demand  on  the  user's  short-term 
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memory  (memory),  help  users  understand  the  structure  of  the  system  (structure),  provide 
feedback  and  error-correction  mechanisms  (workload),  and  accommodate  individual 
differences  (individualization). 

From  an  engineering  point  of  view,  Shneiderman  (1992)  detailed  four  essential 
classes  of  goals  for  interface  design.  They  are  (1)  functionality  to  accomplish  all 
assigned  tasks,  (2)  foolproof  reliability,  availability,  security  and  data  integrity,  (3) 
standardized  styles  and  interface  features  to  ensure  portability  over  multiple  applications, 
and  (4)  development  should  maintain  schedule  and  remain  within  budget.  The  goal  of 
interface  design  not  only  includes  assisting  the  user  in  accomplishing  their  task  of 
controlling  or  influencing  a  target  system,  it  should  also  help  the  user  understand  the 
system. 

In  most  cases,  the  target  system  and  the  user  interface  are  usually  designed  by 
separate  individuals,  who  are  often  from  different  groups  or  organizations.  The  design 
activities  may  take  place  at  different  places  and  at  different  times,  making  effective 
communications  between  the  two  groups  especially  challenging.  The  two  design  teams 
may  have  different  objectives,  constraints  and  conceptual  bases.  Interface  designers  must 
establish  early  contact  with  not  only  future  users  of  the  system,  but  also  designers  of  the 
system  so  that  both  the  "user  model"  and  the  "design  model"  can  be  clearly  identified  and 
correctly  reconciled.  Interface  designers  must  recognize  and  overcome  these  competing 
models  to  successfully  bridge  the  gap  between  the  user’s  mental  model  and  the  system 
designer’s  conceptual  model  of  the  same  target  system  (Norman,  1983). 
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F.  USABILITY  TESTING 


Although  user  testing  with  real  users  is  the  most  fundamental  usability  method, 
other  usability  methods  can  serve  as  good  supplements  to  gather  additional  information 
or  gain  usability  insight  at  a  lower  cost.  User  testing,  combined  with  heuristic  evaluation 
and  other  usability  inspection  methods,  can  provide  better  efficiency  (Nielsen,  1997). 
Heuristic  evaluation  consists  of  finding  usability  problems  in  a  design  by  contrasting  it 
with  a  list  of  established  usability  principles.  Though  this  method  works  very  well  and 
produces  fast  evaluation  results,  there  are  always  some  issues  that  will  be  left 
undiscovered  until  actual  user  testing  (Nielsen,  1997).  In  all  testing,  issues  of  reliability 
and  validity  need  serious  consideration.  Reliability  considers  whether  one  would  get  the 
same  result  if  a  test  were  repeated  and  validity  considers  whether  the  result  actually 
reflects  the  usability  issues  one  wants  to  test. 

1.  Reliability 

Huge  individual  differences  between  test  users  presents  a  reliability  problem  with 
usability  testing.  It  is  not  uncommon  for  the  best  user  to  be  10  times  faster  than  the 
slowest  user,  or  the  top  25  percent  of  the  users  to  be  twice  as  fast  as  the  slowest  25 
percent  of  the  users  (Egan,  1988).  Due  to  this  phenomenon,  one  cannot  conclude  much 
from  observing  a  user  A  using  interface  X  performing  a  certain  task  40  percent  faster 
than  user  B  using  interface  Y.  User  B  could  very  well  just  happen  to  be  slower  than  user 
A.  If  the  test  were  repeated  with  users  C  and  D,  the  result  could  easily  be  exactly 
opposite.  Often  decisions  must  be  made  on  the  basis  of  fairly  unreliable  data,  which 
while  suspect,  is  better  than  having  no  data  at  all.  Standard  statistical  tests  should  be  used 
assess  test  results  and  thereby  indicate  the  significance  and  reliability.  Although  a  0.05 
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significance  level  is  often  used  for  research  studies,  for  practical  development  purposes 
an  0.20  significance  level  might  be  more  attainable  due  to  budget  and  time  restraints 
(Nielsen,  1997). 

2.  Validity 

While  reliability  can  be  addressed  with  statistical  tests,  a  high  level  of  validity 
requires  full  understanding  of  the  test  method,  as  well  as  common  sense.  Typical  validity 
problems  include  using  the  wrong  users,  giving  them  the  wrong  tasks,  or  not  including 
time  restraints  of  social  influences.  Confounding  effects  may  also  lower  the  validity  of  a 
usability  test.  An  example  would  be  testing  the  move  from  a  character  based  user 
interface  to  a  graphical  user  interface  for  an  application.  If,  while  running  this  test,  the 
two  competing  systems  were  running  on  screens  set  to  different  pixel  graphic  displays, 
the  test  results  may  give  a  comparison  between  the  two  size  screens  rather  than  the 
intended  character-based  versus  graphical  user  interfaces  (Nielsen,  1997). 

3.  Test  Users 

The  primary  concern  regarding  test  users  is  that  they  should  be  as  representative 
as  possible  of  the  intended  users  of  the  system.  Sometimes  the  exact  individuals  who  will 
be  using  a  system  can  be  identified.  Typically,  this  case  is  specific  to  a  system  being 
developed  for  an  organization  or  department  within  an  organization.  Another  case  is  a 
system  targeted  at  a  certain  type  of  user,  such  as  lawyers,  secretaries  or  warehouse 
managers.  In  this  case  the  users  can  be  more  or  less  homogeneous,  but  it  is  desirable  to 
select  users  from  as  many  different  locations  as  possible.  For  software  intended  for  the 
general  population,  almost  anyone  can  serve  as  a  test  user.  Lastly,  care  must  also  be 
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taken  to  ensure  that  both  expert,  novice,  and  mid  skill  level  users  are  chosen  (Neilsen, 
1997). 

4.  Test  Tasks 

Test  tasks  should  be  as  representative  as  possible  of  the  uses  to  which  the  system 
will  eventually  be  fielded.  They  should  also  reasonably  cover  the  most  important  parts  of 
the  user  interface,  be  small  enough  to  be  completed  within  the  test  time  limits,  yet  not 
become  trivial.  The  first  task  should  be  extremely  simple  to  guarantee  the  user  an  early 
success  experience  to  boost  morale.  Conversely,  the  last  task  should  provide  the  user 
with  the  feeling  of  accomplishment  (Gaylin,  1986). 

Usability  tests  typically  have  four  stages  (Nielsen,  1997): 

1.  Preparation:  Ensure  the  test  room  is  ready  and  that  the  computer  system  is 
in  the  start  state,  and  that  all  test  materials  and  instructions  are  available. 

2.  Introduction:  Welcome  the  test  user  and  give  a  brief  expalnation  of  the 
purpose  of  the  test,  to  include  computer  setup  if  necessary. 

3.  Running  the  test:  During  the  test  itself,  the  experimenter  should  normally 
refrain  from  interacting  with  the  user.  The  exception  to  this  rule  is  the 
case  where  the  test  users  is  clearly  stuck  and  becoming  quite  unhappy  with 
the  situation.  Caution  must  be  exercised  not  to  help  the  user  too  early. 

4.  Debriefing:  After  the  test,  debrief  the  user  and  get  any  subjective  input  as 
soon  as  possible.  During  the  debrief,  extract  from  the  user  their  input 
regarding  the  test  itself.  Also,  as  soon  as  possible,  ensure  that  all  results  of 
the  test  have  been  collected  and  properly  labeled. 

Consideration  of  the  required  tasks  is  critical  to  properly  setting  up  the  test.  It  is 
important  that  the  test  user  feel  as  if  their  input  is  significant,  beneficial  and  appreciated. 

| 
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5.  Performance  Measurement 

Measurement  studies  for  the  basis  of  much  traditional  research  on  human  factors. 
User  performance  is  almost  always  measured  by  having  a  group  of  test  users  perform  a 
predefined  set  of  test  tasks  while  collecting  time  and  error  data  (Nielsen,  1997).  There  is 
the  potential,  however,  to  measure  something  that  is  poorly  related  to  the  property  which 
is  attempting  to  be  assessed.  Goals  are  generally  abstract,  so  care  must  be  taken  to  break 
them  down  into  components  with  specific  usability  attributes,  which  are  to  be  measured 
by  specific  tasks.  Some  usability  studies  use  qualitative  methods  rather  than  exact 
measurements.  The  focus  of  the  test  could  be  to  determine  which  aspects  of  the  system 
work  well  and  which  are  troublesome,  yet  not  have  any  exact  measures  to  differentiate 
the  two.  To  mine  all  possible  observations  of  qualitative  data  analysis  requires  extensive 
engineering  experience.  Even  inexperienced  usability  engineers,  however,  can  usually 
ascertain  major  qualitative  findings,  as  they  are  more  likely  to  be  readily  apparent 
(Nielsen,  1997). 

G.  SUMMARY 

Naval  Aviation  has  made  significant  strides  in  reducing  the  mishap  rate  in  the  last 
half  century.  These  advances  can  be  attributed  to  more  standardized  investigation  and 
reporting  systems,  the  use  of  systems  engineering  and  the  application  of  human  error 
causation  theory  on  mishap  cause  factors.  Significant  amounts  of  data  are  being 
collected,  cataloged  and  stored  for  future  reference.  Computer  database  systems  make 
the  organization  of  this  data  relatively  simple.  For  Naval  Aviation  to  take  the  next  step, 
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however,  a  robust  system  for  analyzing  stored  data  for  trend  analysis  and  prevention  is 
needed. 

In  an  effort  to  provide  a  more  useful  data  analysis  system,  HFACS  was 
developed,  then  expanded  to  cover  MRMs  in  HFACS-ME.  Proven  to  be  effective  in 
capturing  the  nature  of,  and  relationships  among,  latent  conditions  and  active  failures. 

The  HFACS-ME  taxonomy  was  developed  into  a  laptop  prototype  database  tool, 

MEIMS.  In  a  limited  usability  study,  MEIMS  was  discovered  to  have  great  potential.  To 
fully  attain  its  potential,  however,  MEIMS  must  be  proven  to  be  a  user  friendly  system 
which  fleet  users  will  embrace.  Further  system  development  and  more  rigorous  usability 
testing  is  required.  With  proper  advances,  MEIMS  can  be  especially  beneficial  in 
reducing  aviation  maintenance  related  mishaps. 
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III.  METHODOLOGY 


A.  RESEARCH  APPROACH 

A  desktop  investigating,  reporting  and  analysis  system  focusing  on  maintenance 
error  in  aviation  mishaps  can  greatly  advance  efforts  to  enhance  safety  and  reduce 
mishaps.  This  information  management  system’s  database  is  populated  with  historical 
records  of  aviation  maintenance  mishap  information.  Additionally,  a  cursory  training 
lesson  and  a  hands-on  experience  tool  are  developed  which  assists  the  participants  in  the 
use  of  the  database.  Next,  the  prototype  version  of  the  tool  is  distributed  to  two  sample 
groups;  one  getting  the  training  lesson  and  an  operator’s  manual  and  the  others  getting  the 
training  session,  an  operator's  manual,  and  a  hands-on  experience  session. 

Participants  in  both  groups  are  then  provided  a  prepared  task  list  that  required 
them  to  navigate  through  the  system.  At  the  completion  of  the  task  list,  all  participants 
have  viewed  all  portions  of  the  prototype  system,  and  have  had  the  opportunity  to  form 
an  opinion  on  its  effectiveness.  The  two  groups’  performance  is  evaluated  to  determine 
the  usefulness  of  the  hands-on  experience  session  in  completing  the  assigned  tasks. 
Additionally,  all  participants  complete  an  exit  survey  composed  of  demographic 
background  questions,  opinion  items  of  the  prototype  system  (including  improvements), 
opinion  items  on  the  value  of  the  training  lesson  and  the  hands-on  experience  session. 
They  also  answer  open-ended  questions  on  the  prototype  system,  training  session  and 
hands-on  experience  session. 
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B.  DATA  COLLECTION 


1.  Participants 

Students  attending  the  Naval  Postgraduate  School  (Monterey,  CA)  serve  as 
participants  in  the  study.  Most  of  the  participants  have  recently  come  from  fleet 
department  head  tours  or  embarked  airwing  staffs.  These  aviators  are  from  all  aircraft 
communities  and  within  the  U.S.  Navy  and  Marine  Corps.  All  have  had  some  experience 
with  mishap  information  reports  and  the  data  contained  therein.  They  are  representative 
of  the  type  of  people  that  will  likely  be  the  primary  users  of  the  tool.  The  participants 
were  split  into  two  groups  of  ten.  The  groups  were  distributed  evenly  matching  rank, 
branch  of  service,  type  aircraft  flown  and  years  of  aviation  experience.  One  group 
receives  a  training  lesson  on  the  tool  along  with  the  instruction  manual  and  the  other 
group  gets  the  training  lesson,  instruction  manual  and  an  additional  in  depth  tutorial. 

2.  Apparatus 

The  participants  are  introduced  to  the  Human  Factors  Analysis  and  Classification 
System-Maintenance  Extension  (HFACS-ME)  taxonomy  and  to  the  Maintenance  Error 
Information  Management  System  (MEIMS)  prototype  data  base  tool.  Additionally,  the 
students  have  access  to  three  computer  labs  at  the  School  of  Aviation  Safety  via  login  ID 
and  password  to  a  group  account.  Each  computer  in  the  laboratories  is  a  Pentium  I, 
Windows  2000  operating  system,  with  15-inch  monitor  of  800  x  600  resolution  or  better. 
Each  has  a  full  functioning  prototype  of  the  tool  loaded  onto  it.  Prior  to  logging  on,  both 
groups  are  given  a  cursory  training  lesson  and  instruction  manual  in  the  use  of  MEIMS. 
Additionally,  one  group  is  provided  with  an  additional  in  depth  tutorial  in  HFACS-ME 
taxonomy  and  MEIMS  capabilities.  After  a  participant  gains  access  to  the  computer,  the 
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icon  “MEIMS  Tool”  is  found  on  the  computer  desktop.  Once  the  icon  is  selected,  the 
prototype  opens  using  Microsoft  Access  2000. 

3.  Instrument 

The  participant  completes  a  usability  test  consisting  of  eleven  tasks.  After 
completion  of  the  test,  an  exit  survey  is  constructed  for  completion  by  the  participants 
consisting  of  three  parts:  (1)  demographics  of  the  participant,  (2)  Likert-type  questions 
assessing  feelings  towards  the  prototype  tool  (including  improvements),  the  hands-on 
experience  session  and  tutorial  (as  applicable),  and  (3)  open-ended  items  to  elicit 
subjective  responses.  Participants  receiving  the  tutorial  answer  additional  Likert-type 
questions  and  open-ended  questions  regarding  the  tutorial.  Survey  questions  are  designed 
to  determine  if  the  tool  meets  user  investigation,  reporting,  and  analysis  requirements. 

The  survey  also  queries  the  participants  getting  the  tutorial  in  its  value.  Additionally, 
participant  performance  between  the  two  groups  is  evaluated  to  determine  the 
effectiveness  of  the  tutorial. 

Collection  of  demographic  information  is  accomplished  through  the  participant 
selecting  from  a  list  of  descriptors  (squadron,  experience  with  computers,  software 
familiarity,  and  operating  system  knowledge).  The  Likert-type  questions  use  a  five  point 
rating  scale  with  verbal  anchors:  Strongly  Agree,  Agree,  Neutral,  Disagree,  and  Strongly 
Disagree.  Open-ended  questions  are  included  to  gain  inputs  on  the  overall  impression  of 
the  training  lesson,  prototype  tool,  the  hands-on  experience  session,  the  tutorial  (as 
applicable),  and  recommendations  for  improvement. 
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4.  Procedure 


MEIMS  was  revised  and  updated  to  include  data  derived  from  an  available 
database;  a  sample  group  of  users  will  measure  its  effectiveness.  The  participants  are 
split  into  two  similar  groups.  One  group  is  provided  an  in  depth  tutorial  (See  Appendix 
B)  prior  to  using  MEIMS.  The  perceptions  and  performance  of  both  groups  are  measured 
to  determine  the  value  of  the  MEIMS  and  the  tutorial. 

All  Participants  are  given  an  overview  presentation  of  the  study  consisting  of  a 
projected  computer  demonstration  of  the  prototype  tool  and  distribution  of  materials 
necessary  to  carry  out  the  user  test.  The  overview  consists  of: 

•  Instructions  for  Accessing  the  Prototype  Tool  -  information  to  log  on  and  open  the 
prototype. 

•  MEIMS  Evaluation  (Appendix  C)— a  series  of  planned  navigation  routes  within 
the  prototype  whereby  the  participant  would  be  able  to  view  the  entire  system. 

•  MEIMS  Exit  Survey  (Appendix  C)~ Participants  complete  an  exit  survey 
composed  of  demographic  background  questions,  impressions  of  MEIMS  and  the 
tutorial  (as  applicable)  and  open-ended  question  requesting  participant’s  opinions. 
All  participants  perform  the  above  actions:  access  the  prototype  tool,  navigate  the 

system  using  the  prototype  task  list,  and  provide  feedback  on  the  system  by  completing 
the  exit  survey.  Additionally,  the  group  receiving  the  tutorial  makes  comments  on  its 
effectiveness  and  usefulness.  The  performance  of  the  tutorial  group  is  compared  with  the 
non-tutorial  group  to  determine  the  benefits  of  the  tutorial. 
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C.  DATA  TABULATION 


The  data  is  transcribed  from  the  survey  onto  a  Microsoft  Excel  2000  spreadsheet. 
The  Likert-type  questions,  based  on  a  five-point  scale,  are  coded  into  the  software  (1 
through  5)  corresponding  to  the  respective  anchors  (Strongly  Disagree,  Disagree,  Neutral, 
Agree,  and  Strongly  Agree).  Specific  categories  (MEIMS  is:  Logical,  Easy  to  Navigate, 
Very  Interesting,  Relevant  and  a  Good  Concept)  are  examined  for  significant  differences 
between  the  tutorial  and  non-tutorial  groups.  Open-ended  questions  are  tabulated  by 
theme  and  number  of  responses. 

D.  DATA  ANALYSIS 

The  data  analysis  tools  of  Microsoft  Excel  2000  are  used  to  generate  descriptive 
statistics  including  the  mean,  standard  deviation,  and  range  of  the  collected  data.  The 
performance  of  the  two  groups  is  tabulated,  measured  and  compared.  Independent  t-tests 
are  conducted  on  the  data  to  determine  significant  differences  between  the  tutorial  and 
non-tutorial  groups.  Participant  responses  to  the  open-ended  questions  are  also  explored 
to  assess  the  perceived  value  of  the  hands-on  experience  session,  the  MEIMS  tool  itself 
and  the  tutorial. 
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IV.  RESULTS 


A.  TEST  SAMPLE 

A  usability  test  was  administered  to  twenty  students  at  the  Naval  Postgraduate 
School.  They  were  randomly  split  into  two  groups  of  ten  each.  Many  of  the  participants 
(n=17;  85%)  were  designated  Naval  Aviators  or  Naval  Flight  Officers  and  represented  a 
cross  section  of  the  aviation  commands  that  make  up  the  squadrons  in  the  Navy  and 
Marine  Corps.  One  group  of  ten  received  a  formal  tutorial  prior  to  testing  the  MEIMS 
prototype,  while  the  other  did  not.  One  of  the  non-aviators  was  in  the  tutorial  group,  two 
were  not.  Both  groups  were  directed  to  access  MEIMS  and  complete  an  eleven-item  task 
list.  Following  the  tasks,  individual  participants  were  asked  to  complete  an  exit  survey. 
The  survey  consisted  of  demographic  information,  queries  regarding  their  satisfaction 
with  MEIMS  and  queries  regarding  their  satisfaction  with  the  tutorial,  if  applicable. 

B.  TEST  TASKS 

The  tasks  portion  of  the  test  was  designed  to  not  only  introduce  the  participants  to 
MEIMS  and  exercise  some  of  its  capabilities,  but  also  to  test  the  effectiveness  of  the 
tutorial.  The  tasks  required  the  participant  to  access  all  functional  areas  of  MEIMS.  The 
tasks  started  relatively  simply  but  became  more  complex  as  the  participant  became  more 
familiar  with  MEIMS.  Test  performance  is  summarized  in  Table  2. 
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Table  2:  Test  Task  Performance 


NUMBER  OF  CORRECT  RESPONSES 

TASK 

TUTORIAL 

NON-TUTORIAL 

1 

10 

10 

2 

10 

10 

3 

10 

10 

4 

10 

10 

5 

10 

10 

6 

10 

7 

7 

10 

10 

8 

9 

6 

9 

10 

10 

10 

10 

10 

11 

10 

10 

The  first  task  was  accessing  the  program;  all  participants  (n=20;  100%)  were  able 

to  access  MEIMS  with  no  difficulty.  The  second  task  requested  the  participant’s  opinion 

of  the  main  menu.  A  number  of  participants  from  both  the  tutorial  group  (n=6;  60%)  and 

the  non-tutorial  group  (n=4;  40%)  noted  that  the  tab  order  was  not  sequencing  properly  in 

the  main  menu.  The  third  task  required  the  participant  to  query  a  type  of  aircraft  and 

determine  how  many  mishaps  exist  in  the  database  for  that  type  aircraft.  Secondly,  the 

participant  had  to  identify  and  define  the  level  3  codes.  All  participants  (n=20;  100%) 

were  able  to  correctly  complete  this  task.  The  fourth  task  asked  the  participant  which  of 

the  seven  categories  was  most  useful  and  which  was  least  useful.  Aircraft  model  (n=17; 

85%)  was  considered  the  most  useful,  while  Location  of  Mishap  and  Aircraft  Type  (n=6; 

30%)  were  each  viewed  as  the  least  useful. 

The  fifth  task  asked  the  participants  to  select  their  own  criteria  for  a  multiple 

query.  Some  participants  (n=5;  25%)  created  queries  that  were  too  restrictive  for  the 
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limited  number  of  mishaps  in  the  prototype  database.  Once  they  viewed  the  error 
message  that  their  query  was  too  restrictive,  they  were  able  to  reselect  and  complete  the 
query.  Both  the  tutorial  (n=8;  80%)  group  and  non-tutorial  (n=7;  70%)  group  thought 
this  feature  was  extremely  useful.  The  sixth  task  required  the  participant  to  access  the 
HFACS-ME  Summary  query  and  determine  how  many  mishaps  were  in  the  database. 

All  participants  (n=20;  100%)  correctly  obtained  this  information.  The  participant  then 
had  to  determine  how  many  level  one  Worker  Condition  mishaps  were  in  the  database 
and  how  many  level  two  Medical  conditions  were  in  the  database.  All  of  the  tutorial 
group  (n=10;  100%)  correctly  answered  this  question,  however  not  everyone  in  the  non¬ 
tutorial  group  (n-7;  70%)  correctly  answered  it.  The  seventh  task  asked  the  participant  to 
determine  the  number  of  mishaps  of  another  type  aircraft.  All  participants  (n=20;  100%) 
correctly  determined  the  number  of  mishaps. 

The  eighth  task  required  the  participant  to  determine  the  number  of  Detached 
mishaps  in  the  Mishap  Distribution  -  Location  Report.  In  the  tutorial  group,  almost  all 
(N=9;  90%)  correctly  answered  this  question.  In  the  non-tutorial  group,  over  half  (n=6; 
60%)  correctly  answered  it.  The  ninth  task  asked  the  participant  to  graph  two  type 
aircraft  on  the  X-Axis  versus  HFACS-Level  One  codes  on  the  Y-Axis.  All  participants 
(n=20;  100%)  were  able  make  the  graph.  Some  participants  in  both  the  tutorial  (n=4; 
40%)  and  the  non-tutorial  (n=2;  20%)  groups  noted  that  the  graph  provided  frequency  of 
mishaps,  vice  some  normalized  baseline,  like  percent  of  mishaps  per  type  aircraft.  The 
tenth  task  was  to  input  two  mishaps  into  the  database.  All  members  of  the  both  groups 
(n=20;  100%)  were  able  to  successfully  enter  mishaps  into  the  database.  It  was  noted  by 
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many  participants  in  both  groups  (n=16:  80%)  that  the  database  does  not  adequately  error 
check  the  information  being  input. 

The  eleventh  task  asked  the  participant  to  check  the  Chronological  Listing  of  the 
mishaps  that  had  been  input  into  MEIMS.  Again,  all  the  participants  (n=20;  100%) 
correctly  completed  this  task.  Overall  the  performance  of  the  tutorial  group  was  slightly 
better  than  the  non-tutorial  group  (See  Figure  7).  In  the  tutorial  group  almost  all  (n=9; 
90%)  had  no  errors,  while  in  the  non-tutorial  group  less  than  half  (n=4, 40%)  had  no 
errors. 


0  0  0  0 
Number  of  Errors 


I B  Tutorial  Group  B  Non-Tutorial  Group  j  [ 

Figure  7:  Test  Task  Performance 

The  mean  percentage  of  correct  responses  for  the  tutorial  group  was  99. 1  (SD  = 
2.84,  RNG  =  10),  whereas  the  non-tutorial  group  mean  was  93.7  (SD  =  6.07,  RNG  =  19). 
An  independent  tequai  test  comparing  mean  tutorial  and  non-tutorial  group  performance 
was  conducted.  It  assumed  equal  n,  no  difference  between  group  means,  and 
homogeneity  of  variance.  The  observed  t^  of  2.55  was  significant  at  the  p  =  0.02  level 
for  a  two  tailed  test  (See  table  Al,  Appendix  A).  Given  the  variance  for  the  tutorial 
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group  was  8.1  and  for  the  non-tutorial  group  was  36.9,  a  second  t-test  was  run  for 
unequal  variance  to  reduce  the  chance  of  Type  I  error.  The  results  are  presented  in  Table 
3.  The  observed  tunequa]  of  2.55  was  significant  at  the  0.02  level. 

Table  3:  Task  Performance  Two-Sample  t-Test  Assuming  Unequal  Variance 


T  utorial 

Non-tutorial 

n 

10 

10 

Mean 

99.10 

93.70 

Variance 

8.10 

36.90 

df 

13 

tCrit 

2.16 

tStat 

2.55 

P  two-tail 

0.02 

C.  DEMOGRAPHIC  INFORMATION 

Part  I  of  the  exit  survey  requested  demographic  information  from  the  participants. 
It  established  computer  experience  levels  to  determine  if  computer  or  aviation  experience 
had  an  impact  on  that  participant’s  ability  to  utilize,  or  their  satisfaction  with,  the  MEIMS 
tool.  Demographic  information  is  summarized  in  Table  4. 
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Table  4:  Demographic  Information 


Number  of  Participants 

Demographic 

Tutorial 

Non-Tutorial 

SQDRN  MX 

9 

7 

2  +  YRS  Computer  EXP 

10 

10 

Microsoft  Office 

10 

10 

Lotus  Smart  Suite 

1 

1 

Corel  Office 

0 

1 

Word  Processing 

10 

10 

Presentations 

10 

10 

Graphic  Software 

10 

10 

E-Mail 

10 

10 

Database 

4 

6 

Windows  (3.1-2000) 

10 

10 

Windows  NT 

10 

10 

Macintosh 

1 

0 

Linux 

0 

1 

Question  one  determined  that  9  of  the  tutorial  group  participants  had  been 
members  of  aviation  units  that  performed  squadron  level  maintenance  (n=9;  90%).  The 
other  member  of  the  tutorial  group  was  from  a  non-aviation  background  (n=l;  10%).  In 
the  non-tutorial  group,  7  participants  were  members  of  aviation  units  that  performed 
squadron  level  maintenance  (n=7,  70%),  one  was  a  member  of  a  staff  that  did  not 
perform  maintenance  (n=l;  10%),  and  two  were  from  non-aviation  backgrounds  (n=2; 
20).  Question  two  indicated  that  all  participants  had  at  least  two  years  of  experience 
using  a  computer  (n=20;  100%).  Question  three  determined  that  all  participants  (n  =  20; 
100%)  were  users  of  Microsoft  Office.  Minimal  numbers  in  either  group  had  used  either 
Lotus  Notes  (n  =  2;  20%)  or  Corel/Word  Perfect  (n  =  1;  10%).  Question  four  established 
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a  participant’s  familiarity  with  different  software  applications;  all  participants  (n=20; 
100%)  were  familiar  with  processing,  spreadsheet,  presentation,  and  e-mail.  In  the 
tutorial  group,  four  were  familiar  with  database  applications  (n=4;  40%),  while  in  the 
non-tutorial  group,  that  number  was  6  (n=6;  60%).  Question  five  determined  that  all 
participants  (n=20;  100%)  were  familiar  with  Windows  97/2000,  Windows  NT  operating 
systems,  or  both.  The  MEIMS  tool  was  loaded  on  computers  running  the  Windows  NT 
operating  systems. 

D.  USER  SATISFACTION  WITH  MEIMS  TOOL 

1.  Impressions  of  MEIMS 

Part  II  of  the  exit  survey  requested  the  participant’s  impressions  of  the  MEIMS 
tool  and  its  value  to  Naval  Aviation.  Participants  responded  to  five  statements  using 
Likert  type  responses  selecting  from  one  of  five  responses:  strongly  agree,  agree,  neutral, 
disagree,  and  strongly  disagree.  Values  of  five  through  one  respectively  were  assigned  to 
the  statements.  Additionally,  participants  could  make  subjective  comments  on  any  of  the 
statements. 

Statement  one  asked  whether  or  not  a  participant  found  MEIMS  to  be  presented  in 
a  logical  form.  The  histogram  of  the  frequency  distribution  for  statement  one  is 
presented  in  Figure  8.  The  mean  for  the  tutorial  group  was  4.4  (SD  =  0.52;  RNG  =  2), 
whereas  the  non-tutorial  group  mean  was  4. 1  (SD  =  0.57;  RNG  =  3).  All  participants  in 
the  tutorial  group  agreed  that  MEIMS  presented  the  information  in  a  logical  fashion 
(n=10;  100%),  while  in  the  non-tutorial  group  it  was  less  than  unanimous  (n=9;  90%). 
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Figure  8:  MEIMS  is  in  a  Logical  Form 


Statement  two  asked  about  the  ease  of  navigation  of  the  prototype.  The  histogram 
of  the  frequency  distribution  for  statement  two  is  presented  in  Figure  9.  The  mean  for  the 
tutorial  group  was  4.0  (SD  =  0.5;  RNG  =  3),  whereas  the  non-tutorial  group  mean  was 
3.5  (SD  =  0.97;  RNG  =  4).  A  majority  of  participants  in  both  the  tutorial  group  (n=9; 
90%)  and  the  non-tutorial  group  (n=6;  60%)  agreed  that  MEIMS  was  easy  to  navigate. 


Figure  9:  MEIMS  is  Easy  to  Navigate 


Statement  three  asked  if  the  participants  felt  MEIMS  was  “very  interesting.”  The 
histogram  of  the  frequency  distribution  for  statement  three  is  presented  in  Figure  10.  The 
mean  for  the  tutorial  group  was  4.2  (SD  =  0.63;  RNG  =  3),  whereas  the  non-tutorial 
group  mean  was  3.7  (SD  =  0.82;  RNG  =  4).  Most  participants  in  both  the  tutorial  group 
(n=9;  90%)  and  the  non-tutorial  group  (n=7;  70%)  agreed  that  MEIMS  was  very 
interesting. 


Figure  10:  MEIMS  is  Very  Interesting 


Statement  four  asked  about  the  relevance  of  MEIMS  to  aviation  maintenance 
operations.  The  histogram  of  the  frequency  distribution  for  statement  four  is  presented  in 
Figure  11.  The  mean  for  the  tutorial  group  was  4.6  (SD  =  0.52;  RNG  =  2),  whereas  the 
non-tutorial  group  mean  was  4.5  (SD  =  0.53;  RNG  =  2).  All  participants  agreed  that 
MEIMS  was  relevant  (n=20;  100%). 
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Figure  11:  MEIMS  is  Relevant  to  Maintenance  Operations 

Statement  five  asked  whether  prototype  concept  was  a  good  one.  The  histogram 
of  the  frequency  distribution  for  statement  five  is  presented  in  Figure  12.  The  mean  for 
the  tutorial  group  was  4.6  (SD  =  0.52;  RNG  =  2),  whereas  the  non-tutorial  group  mean 
was  4.5  (SD  =  0.53,  RNG  =  2).  Again,  all  participants  agreed  that  the  concept  of  MEIMS 
was  a  good  one  (n=20;  100%). 


Figure  12:  MEIMS  Concept  is  Good 
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Independent  tequai  tests  comparing  tutorial  and  non-tutorial  responses  to  the  Likert- 
type  questions  were  conducted.  They  assumed  equal  n,  no  difference  between  group 
means,  and  homogeneity  of  variance  (See  Appendix  A).  For  the  Logical,  Navigation, 
Relevance  and  Concept  categories,  no  significant  difference  was  noted.  For  the 
Interesting  category,  an  observed  tequai  of  2.59  was  significant  at  the  p  =  0.02  level  for  the 
two  tailed  test  (See  table  A4,  Appendix  A).  Given  the  variance  for  the  tutorial  group  was 
0.28  and  for  the  non-tutorial  group  was  0.68,  a  second  t-test  was  run  for  unequal  variance 
to  reduce  the  chance  of  Type  I  error.  The  results  are  presented  in  Table  5.  The  observed 
tunequai  of  2.59  was  significant  at  the  0.02  level. 

Table  5:  MEIMS  is  Interesting  Two-Sample  t-Test  Assuming 
Unequal  Variance 


Tutorial 

Non-tutorial 

n 

10 

10 

Mean 

4.50 

3.70 

Variance 

0.28 

0.68 

df 

15 

i 

tCrit 

2.13 

tStat 

2.59 

P  two-tail 

0.02 

2.  Open-ended  Questions 

Part  HI  of  the  exit  survey  contained  three  open-ended  questions  regarding  the 
participants  overall  satisfaction  with  MEIMS.  All  participants  made  constructive 
criticism,  although  overall,  the  response  was  very  positive.  The  comments  indicated  that 
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MEIMS  is  currently  a  good  tool  that  has  the  potential  to  be  extremely  valuable  instrument 
in  the  prevention  of  mishaps  (See  Table  6) 

Question  one  asked  the  participant  to  list  the  most  positive  aspects  of  the 
prototype.  Overall,  the  response  was  positive.  Twelve  participants  commented  to  the 
effect  that  MEIMS  would  make  an  immediate,  positive  impact  on  safety  in  the  fleet. 

Question  two  asked  for  the  most  negative  aspects  of  the  prototype.  Most 
comments  were  problems  or  suggested  improvement  to  the  interface.  Such  items 
included  error  checking  capability  for  data  input,  enlarge  text  boxes  so  the  data  can  be 
completely  read,  line  wrap  the  Contributing  Factors  text  box  so  that  the  factors  are  not  cut 
off  after  one  line,  and  improving  the  tab  function  in  the  main  menu,  improve  the  page  to 
page  navigation.  Numerous  suggestions  were  made  for  “normalizing”  the  graphing 
functions.  Several  participants  suggested  a  “find”  mode  in  reports  to  prevent  paging 
through  them  to  find  the  data  desired.  Responses  produced  no  discemable  difference 
between  the  tutorial  and  non-tutorial  groups  regarding  their  impression  of  MEIMS. 

Familiarity  and  knowledge  of  HFACS-ME  terminology,  however,  was  a  distinct 
problem  for  several  of  the  non-tutorial  group.  Seven  participants  made  some  comment 
regarding  HFACS-ME  terminology.  Most  found  it  hard  to  understand.  Conversely,  none 
of  the  tutorial  group,  to  whom  the  HFACS-ME  terminology  was  thoroughly  explained, 
had  a  negative  comment  regarding  HFACS-ME  terminology. 

Question  three  requested  suggestions  for  changes  to  MEIMS.  Most  participants 
re-iterated  their  previously  mentioned  “negatives.”  One  participant  made  the  suggestion 
that  the  database  should  cross-reference  the  MIR  messages  for  each  mishap.  Seventeen 
made  suggestions  regarding  navigation,  error  checking  and  GUI  interface.  One 
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participant  mentioned  making  MEIMS  available  on  the  World  Wide  Web.  There  were  no 
noteworthy  differences  between  the  tutorial  and  non-tutorial  groups  in  response  to  this 
question. 


Table  6:  Responses  to  Open  Ended  Questions 


Number  of  Responses 

Theme 

Tutorial 

Non-tutorial 

Positive  Impact 

5 

7 

Normalize  Graphs 

4 

2 

Find  Mode  in  Reports 

2 

3 

HFACS  Terminology  Difficult 

0 

7 

Improve  Error  Checking  For  Data  Entry 

9 

6 

Improve  Navigation 

4 

6 

Link  to  MIR  message 

1 

0 

GUI  Improvements 

8 

9 

E.  USER  SATISFACTION  WITH  THE  TUTORIAL 
1.  Impressions  of  the  Tutorial 

Part  IV  of  the  exit  survey  requested  the  participant’s  impressions  of  the  tutorial. 
Participants  responded  to  five  statements  using  Likert-type  responses  selecting  from  one 
of  five  responses:  strongly  agree,  agree,  neutral,  disagree,  and  strongly  disagree.  Values 
of  five  through  one  respectively  were  assigned  to  the  statements.  Additionally, 
participants  could  make  subjective  comments  on  any  of  the  statements  (See  Table  7) 
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Table  7:  Summary  of  Responses  to  Tutorial 


Tutorial 

Strongly  Agree 

Agree 

Neutral 

Disagree  !  Strongly  Disagree 

Beneficial 

3 

6 

1 

0 

0 

Made  Eva!  Easier 

6 

3 

0 

1 

0 

nsnn^m 

5  ! 

5 

0 

0 

0 

Relevant 

8 

2 

0 

0 

0 

m 

6 

4 

0 

0  i 

0 

Statement  one  asked  whether  the  tutorial  was  beneficial.  The  histogram  of  the 
frequency  distribution  for  statement  one  is  presented  in  Figure  13.  The  mean  for  the 
tutorial  group  was  4.2  (SD  =  0.63;  RNG  =  3).  All  but  one  participant  in  the  tutorial  group 
agreed  that  the  tutorial  was  beneficial  (n=9;  90%).  The  participant  that  was  neutral 
commented  that  a  user  could  figure  MEIMS  out  in  “15-30  minutes”  without  the  benefit  of 
the  tutorial. 
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Figure  13:  Tutorial  Is  Beneficial 


Statement  two  asked  if  the  tutorial  made  the  usability  test  easier  to  complete.  The 
histogram  of  the  frequency  distribution  for  statement  two  is  presented  in  Figure  14.  The 


48 


mean  for  the  tutorial  group  was  4.54  (SD  =  0.93;  RNG  =  4).  Almost  all  of  the 
participants  in  the  tutorial  group  (n=9;  90%)  considered  the  tutorial  to  be  beneficial.  The 
participant  that  disagreed  restated  the  reason  given  in  the  item  above. 
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Figure  14:  Tutorial  Made  Evaluation  Easier 

Statement  three  asked  if  the  participants  felt  the  tutorial  was  interesting.  The 
histogram  of  the  frequency  distribution  for  statement  three  is  presented  in  Figure  15.  The 
mean  for  the  tutorial  group  was  4.5  (SD  =  0.53;  RNG  =  2).  All  participants  in  the  tutorial 
group  found  it  to  be  interesting  (n=10;  100%). 
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Figure  15:  Tutorial  is  Interesting 

Statement  four  asked  about  the  relevance  of  tutorial  to  MEIMS.  The  histogram  of 
the  frequency  distribution  for  statement  four  is  presented  in  Figure  16.  The  mean  for  the 
tutorial  group  was  4.8  (SD  =  0.42;  RNG  =  2).  All  participants  agreed  that  the  tutorial 


was  relevant  to  MEIMS  (n=10;  100%). 


Figure  16:  Tutorial  Relevant  to  MEIMS 
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Statement  five  asked  whether  the  concept  of  a  tutorial  was  a  good  one.  The 
histogram  of  the  frequency  distribution  for  statement  five  is  presented  in  Figure  17.  . 
The  mean  for  the  tutorial  group  was  4.6  (SD  =  0.52;  RNG  =  2).  Again,  all  participants 
agreed  that  the  tutorial  concept  was  a  good  one  (n=10;  100%). 
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Figure  17:  Tutorial  Concept  is  Good 

2.  Open-ended  Questions 

The  last  portion  of  the  exit  survey  contained  three  open-ended  questions  regarding 
the  participants  overall  satisfaction  with  the  tutorial.  The  participants  were  asked  about 
the  positive  and  negative  aspects  of  the  tutorial.  Lastly,  the  participants  were  encouraged 
to  suggest  changes  to  the  tutorial.  The  comments  made  were  overall  positive.  Some 
constructive  criticism  was  made,  aimed  at  making  the  tutorial  better. 

Question  one  asked  the  participant  to  list  the  most  positive  aspects  of  the  tutorial. 
Comments  referred  to  the  tutorial  as  “comprehensive,”  “focused  on  the  user,”  and 
“providing  the  necessary  background”  to  take  the  test.  One  participant  stated  that  the 
tutorial  was  thorough  without  running  too  long. 
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Question  two  asked  for  the  most  negative  aspects  of  the  tutorial.  There  were 
fewer  negative  comments  than  positive  ones.  They  included  deleting  the  historical 
references  in  the  beginning  of  the  tutorial,  making  the  slides  “less  busy,”  and  too  much 
redundancy  of  the  HFACS-ME  codes. 

Question  three  requested  suggestions  for  changes  to  the  tutorial.  There  was  only 
one  comment  made  regarding  changes  to  the  tutorial  (n=l;  10%).  It  was  suggested  that 
the  tutorial  should  be  self-study,  providing  the  participant  either  the  slides  (with  notes),  or 
on  a  desktop  from  which  to  view  the  tutorial. 
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V.  SUMMARY,  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  SUMMARY 

Naval  Aviation  is  in  the  unenviable  position  of  increased  demand  for  high  levels 
of  operational  readiness,  equipment  availability,  and  personnel  training  within  a  climate 
of  reduced  budgets.  The  notion  of  doing  more  with  less  is  an  everyday  reality.  To  meet 
this  demand,  asset  preservation  is  vitally  important,  as  funding  for  replacements  does  not 
appear  likely  in  the  near  future.  In  this  fiscally  strained  environment,  the  costs  of  Naval 
Aviation  mishaps,  in  terms  of  mission  capability,  operational  readiness,  and  material 
status,  are  too  high.  Reducing  the  number  and  rate  of  mishaps  occurrence  is  paramount 
to  the  success  of  Naval  Aviation.  Furthermore,  reducing  “avoidable”  mishaps,  those 
involving  human  error,  is  an  absolute  must.  Human  error  in  aviation  mishaps  must  be 
identified,  and  appropriate  intervention  strategies  developed,  aimed  at  reducing  the 
causes  of  those  errors.  Conventional  accident  causation  theories  suggest  that  accidents 
are  the  result  of  a  complex  combination  of  errors.  Naval  Aviation  must  target 
intervention  strategies  in  key  areas  to  effectively  combat  human  error. 

Initial  efforts  were  made  solely  in  the  area  of  aircrew  error  reduction,  and  a  robust 
classification  taxonomy,  Human  Factors  Analysis  and  Classification  System  (HFACS) 
was  developed.  It  was  soon  realized  that  to  meet  the  goals  in  mishap  reduction.  Naval 
Aviation  had  to  look  beyond  aircrew  error  and  study  maintenance  human  error.  To  that 
end  a  Maintenance  Extension,  HFACS-ME  was  built.  This  expanded  taxonomy  proved 
to  be  an  acceptable  method  for  classifying  human  factors  in  maintenance  mishaps.  Using 
HFACS-ME  to  study  mishap  causes,  a  clear  picture  of  the  human  factors  contributing  to 
mishaps  can  be  drawn. 
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To  fully  utilize  the  HFACS-ME  taxonomy,  a  prototype  database  tool. 
Maintenance  Error  Monnation  Management  System  (MEIMS)  was  created.  It  enables 
better  access  to  the  data,  query,  graphing  and  reporting  functionality,  and  the  ability  to 
add  new  information  to  the  database.  In  an  initial  usability  study,  MEIMS  was 
determined  to  be  a  valuable  tool  in  the  effort  to  reduce  human  error  in  maintenance.  A 
MEIMS  revision  was  completed,  updating,  and  improving  it,  and  targeting  many  of, the 
issues  detailed  in  the  initial  usability  test.  Additionally,  a  users  tutorial  was  built  to  assist 
the  targeted  user  in  the  introduction  to  MEIMS.  This  study  investigated  the  validity  of 
the  MEIMS  tool  and  the  tutorial.  MEIMS  is  developing  into  a  robust  tool  not  only  for  the 
analysis  of  mishap  causal  factors,  but  also  an  effective  weapon  in  the  development  of 
prevention  programs  tailored  to  human  error  in  maintenance  mishaps.  Its  positive  impact 
on  the  fleet  will  be  immediate  and  far-reaching. 


B.  CONCLUSIONS 

The  participants  of  the  study  were  overwhelmingly  supportive  of  MEIMS  and  the 
tutorial.  Results  from  the  usability  test  and  the  Likert-type  responses  indicate  that  the 
tutorial  was  beneficial  and  made  the  test  easier.  Participants  in  the  tutorial  group 
performed  better  in  the  tasks  portion  than  the  non-tutorial  group.  Based  upon  the 
independent  t-tests  conducted,  there  was  a  significant  performance  difference  between 
the  two  groups,  with  the  tutorial  group  performing  better.  The  response  to  the  Likert-type 
questions  did  not  indicate  a  significant  difference  between  the  tutorial  and  non-tutorial 
groups  (as  determined  by  independent  t-test)  regarding  MEIMS  being  logical,  easy  to 
navigate,  relevant  to  maintenance  operations  and  a  good  concept.  Regarding  MEIMS 
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being  interesting,  an  independent  t-test  indicated  a  significant  difference  between  the 
tutorial  and  non-tutorial  groups,  with  the  tutorial  group  finding  MEIMS  more  interesting. 
This  data  indicates  an  increasing  interest  level  with  increased  exposure  and  knowledge  of 
MEIMS.  The  tutorial  group’s  support  for  MIEMS  was  more  enthusiastic.  Although  both 
groups  strongly  advocated  MEIMS  relevance  to  maintenance  operations  and  endorsed  it 
as  a  good  concept,  the  tutorial  group  was  more  positive  in  supporting  it.  The  participants 
of  the  tutorial  provided  excellent  feedback  and  also  solidly  endorsed  it  as  a 
complimentary  link  with  MEIMS. 

All  participants  made  valuable  comments  and  suggestions.  Numerous  previously 
unrecognized  GUI  deficiencies  were  identified.  Additionally,  data  error  checking, 
graphing  and  reporting  anomalies  were  detailed.  Most  beneficial,  however,  were  the 
suggestions  to  make  MEIMS  a  better  product  for  the  fleet.  Specifically  linking  date- 
time-groups  of  mishap  investigation  reports  to  MEIMS  was  an  especially  astute 
suggestion.  Also,  suggesting  MEIMS  for  the  World  Wide  Web,  a  pre-planned 
progression  for  future  iterations,  was  certainly  perceptive. 

The  potential  of  MEIMS  is  expansive.  MEIMS  should  be  expanded  to  include 
other  services,  allies  and  NATO  mishap  information  to  broaden  the  scope  from  which 
data  is  drawn.  Efforts  are  ongoing  to  make  a  variation  of  MEIMS  that  would  include 
civilian  and  commercial  aviation.  That  would  enable  military  users  an  even  larger  pool 
of  information  from  which  to  draw  data  and  will  be  helpful  in  providing  insights  and 
parallels  between  military  and  civihan/commercial  aviation  safety.  MEIMS  should  soon 
be  available  on  the  World  Wide  Web.  It  will  be  an  excellent  tool  for  users,  from  Naval 
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Aviation’s  highest  executive  levels  down  to  squadron  work  centers,  to  derive  information 
and  to  provide  the  necessary  tools  to  prevent  maintenance  related  mishaps. 

C.  RECOMMENDATIONS 

For  any  software  tool  to  be  effective,  it  must  be  considered  easy  to  use  and 
effective.  That  notion  is  especially  critical  in  a  tool  which  the  fleet  is  expected  to  use.  If 
it  is  not  easy  to  use  and  effective,  it  will  remain  on  the  shelf.  To  that  end,  the  following 
recommendations  are  made  for  the  MEIMS  tool: 

•  Include  the  tutorial  as  a  part  of  MEIMS,  upgrading  as  necessary 

•  Provide  better  navigation  from  MEIMS  functions  to  the  main  menu 

•  Improve  GUI  anomalies  including  the  tab  function  on  the  main  menu, 
improving  text  boxes  so  that  all  data  can  be  read  by  the  user 

•  Improve  the  query  causal  factors  to  add  text  wrap,  so  mishap  information 
will  not  be  cut  off. 

•  Provide  better  error-checking  during  data  input 

•  Add  a  find  mode  in  the  reports  function  to  enable  the  user  to  quickly 
access  the  desired  data  without  paging  through  the  report. 

•  Normalize  graph  presentations  to  provide  a  “weighted”  view'  of  data 

•  Provide  more  sub-menus  to  expedite  multi-layer  functionality 

•  Increase  the  volume  of  the  database  to  include  more  mishaps 

•  Provide  a  link  to  the  date-time-group  of  the  Mishap  Investigation  Report 

•  Make  MEIMS  and  the  tutorial  available  over  the  World  Wide  Web 


X 
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MEIMS  is  already  a  robust  tool  for  analyzing  and  reporting  data  regarding 
maintenance  human  error.  Making  the  above  improvements  will  make  it  an  even  better 
tool.  Once  it  is  available  on-line  with  the  proper  training,  it  should  be  a  well-used  tool  in 
the  effort  to  preserve  lives,  material  and  readiness. 
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APPENDIX  A 


T-TEST  RESULTS  FOR  MEEMS  EVALUATION  AND  PART  II  OF  EXIT 

SURVEY 


1.  MEIMS  TASK  PERFORMANCE 

The  following  is  the  complete  t-test  results  from  the  MEIMS  evaluation  assuming 
equal  n  and  variance. 

Table  Al:  Test  Task  Performance  Two-Sample  t-Test 


Tutorial 

Non-tutorial 

n 

10 

10 

Mean 

99.10 

93.70 

Variance 

8.10 

36.90 

Pooled  Variance 

22.50 

df 

18 

tCrit 

2.10 

tStat 

2.55 

P  two-tail 

0.02 

2.  LIKERT-TYPE  QUESTION  RESPONSE 

The  following  are  the  complete  t-test  results  from  each  of  the  Likert-type 
questions  from  Part  II  of  the  Exit  Survey.  The  t-tests  assume  equal  n  and  variance. 


Table  A2:  MEIMS  is  in  a  Logical  Form  Two-Sample  t-Test 


Tutorial 

Non-tutorial 

n 

10 

10 

Mean 

4.40 

4.10 

Variance 

0.27 

0.32 

Pooled  Variance 

0.29 

i 

df 

18 

tCrit 

2.10 

tStat 

1.24 

P  two-tail 

0.23 
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Table  A3:  MEIMS  is  Easy  to  Navigate  Two-Sample  t-Test 


— 

Non-Tutorial 

n 

10 

10 

Mean 

4.00 

3.50 

Variance 

0.22 

0.94 

Pooled  Variance 

0.58 

df 

18 

tCrit 

2.10 

tStat 

1.46 

P  two-tail 

0.16 

Table  A4:  MEIMS  is  Very  Interesting  Two-Sample  t-Test 


Tutorial 

Non-tutoria! 

n 

10 

10 

Mean 

4.50 

3.70 

Variance 

0.28 

0.68 

Pooled  Variance 

0.48 

df 

18 

tCrit 

2.10 

tStat 

2.59 

P  two-tail 

0.02 
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Table  A5:  MEIMS  is  Relevant  to  Maintenance  Operations  Two-Sample 

t-Test 


Tutorial 

Non-tutorial 

n 

10 

10 

Mean 

4.60 

4.50 

Variance 

0.27 

0.28 

Pooled  Variance 

0.27 

df 

18 

tCrit 

2.10 

tStat 

0.43 

P  two-tail 

0.67 

Table  A6:  MEIMS  Concept  is  Good  Two-Sample  t-Test 


Tutorial 

Non-tutorial 

n 

10 

10 

Mean 

4.90 

4.80 

Variance 

0.10 

0.18 

Pooled  Variance 

0.14 

df 

18 

tCrit 

2.10 

tStat 

0.60 

P  two-tail 

0.56 
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APPENDIX  B 


MEIMS  TUTORIAL  PRESENTATION  WITH  NOTES 

Slide  1 


Maintenance  Error  Information 
Management  System  (MEIMS) 


Human  Factors  Accident 
Classification  System- 
Maintenance  Extension 


LCDK  Mike  "Cracker"  McCracken 
NPS  ITM  Curriculum 
F-I4  RIO 


(Ensure  that  the  students  have  copies  of  these  presentation  slides  to  use  as  a 
reference  during  the  presentation) 

Welcome  to  the  Maintnenancee  Error  Information  Management  System  (MEIMS) 
brief.  This  prototype  tool  is  built  upon  the  Navy’s  HFACS-ME  (Human  Factors 
Accident  Classification  System-  Maintenance  Extension)  taxonomy.  This 
presentation  will  detail  the  HFACS-ME  model  in  detail,  explain  terminology  and 
provide  details.  Please  pay  close  attention  and  ask  questions  as  necessary.  After 
explaining  the  HFACS-ME  taxonomy,  the  MEIMS  tool,  built  upon  that  taxonomy, 
will  be  briefed  and  demonstrated. 
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Slide  2 


Naval  Aviation  Flight  Mishap  Rate 


Naval  aviation  has  become  considerably  safer  over  the  last  few  decades,  but  there  is 
still  a  need  for  improvement. 

This  chart  shows  the  significant  reductions  in  Naval  Aviation  mishap  rates  (per  100,000 
flight  hours)  since  1950. 

Improvements  in  Naval  Aviation  safety  have  primarily  focused  on  engineering  and 
administrative  controls.  They  have  included  (but  have  not  been  limited  to)  design 
improvements  (angled  carrier  decks),  the  creation  of  the  Naval  Safety  Center,  improved 
maintenance  programs  (NAMP),  training  (FRS,  NATOPS),  local  safety  programs,  and 
system  safety  designated  aircraft. 

The  Mishap  rate  has  actually  been  cut  in  half  each  decade  since  1950,  HOWEVER,  there 
are  still  many  Class  A  mishaps  each  year  (155  total  Class  A’s  for  FY  95-99)  which  result 
in  unacceptable  loss  of  life,  disabling  injuries,  and  hundreds  of  millions  of  dollars  in 
damages.  So,  again,  improvements  are  necessary. 


FY  95-99 

Total  Navy/Marine  Aviation  Class  A  Mishaps 
Right  Hours  Right  Mishaps  Rate 

FY95 

1,656,450 

34 

2.05 

FY96 

1,650,026 

36 

2.18 

FY97 

1,523,507 

27 

1.77 

FY98 

1,518,109 

36 

2.37 

FY99 

1,527,186 

22 

1.44 

FY95-99 

7,875,278 

155 

1.97 
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Slide  3 


Engineering  &  Administrative  Controls 
have  Impacted  Hardware  Reliability,  but.... 


Engineering  and  administrative  controls  have  significantly  impacted  hardware  reliability  (the 
aircraft/machinery  are  more  reliable),  but  mishaps  still  occur.  Why? 

This  chart  shows  that  over  the  last  three  decades,  the  material  factors  that  cause  mishaps  (e.g.,  machine 
reliability)  have  steadily  decreased.  Human  error  has  also  decreased  (with  training,  proper  supervision, 
etc.),  but  at  a  slower  rate  so  that  it  has  resulted  in  a  higher  percentage  of  the  factors  that  cause  mishaps.  In 

fact,  the  percentage  of  human  errors  in  mishaps  is  now  increasing. 

In  1999,  there  were  22  Class  A  Flight  Mishaps;  17  had  human  error  involvement  (77%). 

Again,  there  were  155  Navy/Marine  Class  A’s  for  FYs  95-99! 
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than  100%) 


To  further  reduce  the  mishap  rate,  we  must  identify  and  correct  the  causes  of  mishaps. 


Human  error  is  involved  in  the  majority  of  the  determined  causa!  factors  in  this  chart,  with  material  failures 
listed  in  39%  of  the  mishaps.  Human  error  preventative  measures  have  primarily  been  concentrated  on 
aircrew  and  supervisory  issues  (e.g.,  training,  SOPs).  However,  to  reach  the  Human  Factors  Quality 
Management  Board’s  goal  of  reducing  mishaps  by  50%,  we  must  also  focus  on  maintenance  errors. 
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Slide  5 


We  have  just  reviewed  some  statistics  on  major  mishaps;  however,  MAJOR  ACCIDENTS  ARE 
ONLY  THE  TIP  OF  THE  ICEBURG. 

To  make  significant  improvements  in  aviation  maintenance  error,  we  must  thoroughly  analyze  the  major 
mishaps.  Fortunately,  major  accidents  (Class  A  mishaps)  are  relatively  rare  events.  However,  this  low 
mishap  rate  poses  a  new  problem.  There  is  simply  not  enough  information  available  from  Class  A  mishaps 
to  conduct  an  effective  trend  analysis. 

Solution:  For  every  Class  A  mishap,  there  are  numerous  Class  B.  Class  C  and  hazard  reports  which  have 
similar  cause  factors.  We  must,  therefore,  thoroughly  investigate  the  “minor  accidents”  as  well  as  the 
“majors”  to  discover  these  causes. 

Heinrich  Ratio 

The  relatively  few  number  of  catastrophic  accidents  are  the  “tip  of  the  iceberg”.  For  every  major  mishap, 
there  are  10  less  serious  accidents,  30  incidents,  and  600  hazardous  acts.  The  circumstances  (cause  factors) 
which  raise  the  severity  of  the  accident  are  identifiable  in  all  levels  of  accidents  through  adequate 
investigation. 

Source: 

Heinrich,  H.W.,  Petersen,  D.  &  Roos,  N.  (1980).  Industrial  Accident  Prevention:  A  Safety  Management 
Approach  (5th  ed.)  New  York:  McGraw-Hill.  Retrieved  from  http://hfskyway.faa.gov/uaining_toc.htm 
(MRM  training  site). 
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Slide  6 


The  Costs  of  Navy 
Maintenance  Error  Mishaps 

(cost  per  year  in  1997  dollars) 


Class  A  Mishaps 
Class  B  Mishaps 
Class  C  Mishaps 


$85.0  Million 
$3.5  Million 
$3.8  Million 


The  Costs  of  Naw  Maintenance  Error  Mishaps. 

The  costs  are  calculated  from  the  FY90-97  Maintenance  Related  Mishaps  and  adjusted  to  1997 
dollars. 

NOTE:  This  only  includes  component  replacement  costs  and  man  hours  to  restore  the  aircraft  in  operating 
condition.  It  does  not  include  personal  injury,  loss  of  life,  property  damages  or  liability. 

Class  A  S85.0  million/year 
Class  B  $3.5  million/year 

Class  C  $3.8  million/year 

Source:  Aircraft  Maintenance  Human  Factors  PAT  report  to  the  Safety  QMB,  27  June  1998. 
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Slide  7 


Human  Factors 
Quality  Management  Board 


Recommendations 
for  Intervention 


Brainstorming 


^Command 

Safety 

Assessment 

OK,  so  we  must  investigate  all  mishaps.  Are  there  better  or  different  methods  of  investigation?  Has 
anything  been  overlooked? 

Occasionally,  an  accident  is  so  unique  and  overwhelming  that  it  sends  shock  waves  throughout  the 
industry'.  These  “watershed  events”  provide  a  change  in  focus  and  are  major  catalysts  for  change.  One  of 
the  more  famous  “watershed  events”  was  the  1996  Nashville,  Tennessee  F-14  mishap.  This  mishap  had  no 
material  factors.. .it  was  a  human  error  mishap. 

Following  this  mishap,  the  Commander,  Naval  Air  Forces  Pacific  (COMNAVAIRPAC)  empanelled  a 
Human  Factors  Quality  Management  Board  (HFQMB)  to  identify  human  factors  threats  and  develop 
intervention  methods  to  combat  them. 

The  HFQMB  determined  that  Human  Factors  was  the  key  to  reducing  the  mishap  rate  to  1.0  (the 
desired  50%  reduction  per  decade  with  current  rates  stagnate  near  2.0) 

The  HFQMB  adopted  a  three-prong  approach  to  study  human  errors  in  mishaps: 

Mishap  data  analysis  to  identify  hazards  and  risks 

Benchmarking  (of  other  military  services,  NATO  partners,  private  industry,  etc.)  to  uncover 
best  practices  and  find  process  improvements 
Developing  a  Command  Safety  Assessment 

The  QMB's  initial  efforts  were  primarily  directed  at  aircrew  errors,  but  the  QMB  later  directed  a 
Human  Factors  in  Maintenance  and  Material  (HFAMM)  Process  Action  Team  (PAT)  to  address 
maintenance  errors,  as  welL 
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Slide  8 


HFACS-  Maintenance  Extension 


Management  Conditions  _T\ 
Working  Conditions 


HFACS  was  developed  to  identify  factors  that  relate  to  aircrew^  error-  The  Maintenance  Extension 
(HFACS-ME)  was  added  to  analyze  human  error  in  maintenance  related  mishaps.  So  how  does 
HFACS  identify  the  factors  specifically  related  to  aviation  maintenance  accidents? 

The  “Maintenance  Extension5'  taxonomy  for  HFACS  was  developed  to  farther  classify  causal  factors  that 
contribute  to  maintenance  related  mishaps.  This  addition  to  HFACS  consists  of  four  broad  human  error 
categories: 

Management  Conditions  (latent) 

Working  Conditions  (latent) 

Maintainer  Conditions  (latent) 

Maintainer  Acts  (active) 

But  does  it  work?  YES!  HFACS  has  been  applied  effectively  to  the  analysis  of  previous  Navy  mishaps. 
Additional  causal  factors  were  identified,  and  more  importantly,  the  classification  or  coding  of  these  factors 
was  incorporated  into  an  already  extensive  database  of  accidents  and  incidents.  Standardized  coding  allows 
identification  of  trends  throughout  the  various  accidents  and  will  enable  analysts  to  offer  further  prevention 
strategies. 

Most  important:  This  DOES  NOT  require  a  reinvestigation  of  the  mishap.  It  can  be  accomplished, 
with  minimal  training,  by  investigators  or  analysts  simply  coding  the  narratives  and  causal  factors  that 
previously  were  reported. 

Due  to  its  success: 

-  HFACS  has  been  adopted  for  inclusion  into  the  Naval  Aviation  Safety  Program  (  OPNAV  3750.6R) 

-  It  has  influenced  policy  changes  to  the  Navy’s  Maintenance  Program  (4790  series) 

-  And  is  being  used  to  further  the  effectiveness  of  Naval  Aviation’s  methods  of  accident  investigation, 
reporting,  and  database  entry'. 
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Slide  9 


Error  Categories  of  HFACS  Framework 


First  Order 

Second  Order 

Third  Order 

Management 

Conditions 

Organisational 

-  Inadequate  Processes 
.  Inadequate  Resources 

-  Inadequate  Documentation 

-  Inadequate  Design 

Supervisory 

-  Inadequate  Stxpereisjoa 

-  Supervisory  Misconduct 

-  Inappropriate  Operations 

-  UBCorrected  Problem 

Maintainer 

Conditions 

Medical 

•  Mental  State 

.  Physical  State 

-  Limitation  . 

Crete  Coordination. 

-  Communication 

-  Assertweaess 

-  Adap  tabih^.TlexibilitY 

Readiness 

-  Trairing/Preparatien 

*  CcrSBc  aia* n/Q«a!Uj5ca^  » 

,•*  Infringement . 

Working 

Conditions 

Environment 

-  ZoglsSn^'Iagkt 

-  Weasker/Exposure 

-  Environmental  Hazards 

Equipment 

-  Damage&TJnserviced 

*  Unavaalabk/Inappiopriate 

-  Dated'Ditcertifted 

Workspace 

-  Confuting 

-  Obstructed 

-  Inaccessible  -  :  - 

Maiataa»er&cts 

Error 

-  Altcnda 

-  SksH’Teckzdque 

-  Judgment^ecssMfn-Maldng 

.  Knowledg«*’<Me  Based 

Violation 

-Soicfcmc  ■ 

-  Infraction 

-Exceptional  / 

*  Flagjaut  ■ 

The  HFACS-ME  taxonomy  also  includes  three  orders  of  error;  first,  second,  and  third  that  reflect  a 
macro  to  micro  perspective. 

The  levels  provide  sufficient  description  to  develop  a  searchable  database  and  are  useful  for  distinguishing 
supervisory  areas  of  action.  For  example,  a  First  Order  category  could  be  briefed  at  the  Executive  level,  a 
Second  Order  at  the  Vice  President/Senior  Manager  levels,  and  the  Third  Order  can  be  corrected  by  Safety 
Officers  and  front  line  supervisors. 

An  Example:  Maintainer  Conditions  (First  Order),  Readiness  (Second  Order),  and 
Qualification/Certification  (Third  Order) 
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HFACS  -  Maintenance  Extension  Model 


This  model  provides  a  realistic  example  of  the  interaction  between  the  Error  Categories. 

Supervisory  Conditions,  Maintainer  Conditions,  and  Working  Conditions  may 
INDEPENDENTLY  or  COLLECTIVELY  affect  the  Maintainers  Actions. 

The  Unsafe  Maintainer  Act  which  follows  may  either: 

(1)  Lead  directly  to  a  mishap  or  injury  (Example:  a  maintainer  runs  a  forklift  into 
the  side  of  an  aircraft  and  damages  it),  or 

(2)  Become  a  latent  Maintenance  Condition,  which  the  aircrew  would  have  to 
deal  with  on  take-off,  in-flight,  or  on  landing.  (Example:  an  improperly  rigged  landing 
gear  that  collapses  on  touchdown  or  an  over-torqued  hydraulic  line  that  fails  in  flight  and 
causes  a  fire) 

It  is  important  to  note  that  Supervisory  Conditions  related  to  design  for  maintainability, 
prescribed  maintenance  procedures,  and  standard  maintenance  operations  could  be 
inadequate  and  also  lead  directly  to  a  Maintenance  Condition. 

Please  note  that  this  model  utilizes  only  the  First  Order  Error  Categories  (again, 
shown  in  red).  So  how  does  the  Second  Order  Error  Categories  affect  this 
relationship?  (quickly  go  to  next  slide) 
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The  Second  Order  of  Error  Categories  (shown  in  Blue)  is  simply  a  further  breakdown,  or  description,  of  the 
First  Order  Error  Category  (so  additional  relationship  models  are  not  necessary')-  If  we  look  back  a  couple 
of  slides*.. (quickly  go  to  next  slide)... 
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...the  incorporation  of  the  Second  Order  Error  Categories  would  look  like  this. 

Supervisory  Conditions  are  established  at  the  Organizational  levels  (outside  your  command)  or  at  your 
Squadron  level. 

Working  Conditions  that  affect  one  or  more  maintainors  include  the  Environment,  Equipment,  and 
Workspace  limitations. 

Maintainer  Conditions  directly  impact  an  individual  maintainer  and  may  include  Medical,  Crew 
Coordination,  or  Readiness  factors. 

Maintainer  Acts  can  either  be  Errors  or  Violations. 

(NOTE:  Multiple  Error  Categories  may  be  factors  in  mishaps.  For  example,  both  Organizational 
and  Squadron  practices  may  combine  to  form  Supervisory  Conditions  that  lead  to  mishaps,  or  a 
combination  of  Equipment  and  Workspace  conditions  may  promote  an  unsafe  Working  Condition.) 

Now,  as  you  probably  surmised,  the  Third  Order  Error  Categories  are  simply  a  further  breakdown 
of  the  Second  Order  Error  Categories  shown  on  this  slide. 
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First  Order 


Supervisory 

Conditions 


Work:n§ 

Conditions 


Maintainer 

Conditions 


Maintains  r  Acts 


irg  animations! 


Squadron 


Equipment 


Error  Categories  of  HFACS  Framework 


Third  Order 


a!  -  Hazardous  Operations  -  Inadequate  Design  -  Inadequate  Documentation 

-  Inadequate  Resources  -  Inadequate  Processes 

-  Inadequate  Supervision  -  Inappropriate  Operations  -  Uncorrected  Problem 
*  Supervisory  Misconduct 


-  Lights  nought 
Hazards 

-  Darnaqed 


-  Weather/Exposure 


-  Oated/Uncertitied 


Medical  -  Mental  State 

Crew  Coordination  -  Communication 

Adapt  abiiity/riexibility 

Re-adless 

-  T rauni nq/Prepa ration 


-  Attention 
Based 

*  Ski; I  Based 


-  Hhysica!  State 


-  Assertiveness 


rertificatiorrOuahdcation  -  IntVinqement 


* Memory 

-  Jucgm  ent'Becisi  on-Making 


-  Khcwiedqe/Rule 


Here  again,  are  the  Third  Order  Error  Categories  (now  displayed  in  Green). 

The  most  effective  way  to  define  and  display  the  relationship  of  First,  Second,  and  Third  error 
categories  is  to  examine  each  First  Order  Category  independently,  (quickly  go  to  next  slide) 
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Suoervisorv  Conditions 


1  Hazardous  Operational 


Failed  to  Provide  Guidance 
Poor  Risk  Management 
Operational  Doctrine  Errors 
Failed  to  Provide  Training 


1  Inadequate  Docamentatioa  j 


No/Poor  Documentation 
Documents  not  Updated 
Documents  unusable  (CDs) 


Third 

Orders 


1  Inadequate  Resources! 


Improper  Manning 
Lack  of  Funding 
Lack  of  Parts/Equipment 


Purchasing  Failure 
Deficiency  not  Corrected 
Modified  Equipment 


[inadequate  Processes] 


Fail  to  Enforce  Regulations 
Failure  to  Provide  Oversight 
Failed  to  Track  Performance 


Under  Supervisory  Conditions,  we  have  both  Organizational  and  Squadron  Second 
Order  Error  Categories. 

We  will  start  with  the  Organizational  Category. 

(Facilitator  note:  Read  each  term  and  its  example  below  and  then  mention  some  of 
the  additional  examples  listed  in  each  box  on  the  slide.  Encourage  discussion  from 
students) 

Under  the  Organizational  Second  Order,  we  have  the  following  Third  Order  Error 
Categories: 

Hazardous  Operations  An  engine  that  falls  off  of  a  stand  during  a  change  out  evolution 
due  to  an  unforeseen  hazard  of  a  high  seas  state 

Inadequate  Design  The  poor  layout  of  system  components  that  do  not  permit  direct 
observation  of  maintenance  being  perfonned 

Inadequate  Documentation  A  manual  omits  a  step  in  a  maintenance  procedure,  such 
as  leaving  out  an  o-ring  that  causes  a  fuel  leak 

Inadequate  Resources  Insufficient  funding,  manpower,  tools,  parts  and  equipment  to 
perform  maintenance  effectively  and  safely 

Inadequate  Processes  Limitations  in  teamwork,  communication,  and  directives  within 
and  between  organizations 

(NOTE:  The  descriptions  given,  and  the  examples  shown  on  the  slide,  are  just  that . 

. .  examples.  The  purpose  of  this  presentation  is  to  establish  a  knowledge  and 
understanding  of  the  HFACS-ME  Categories,  not  restrict  their  use  to  only  a  handful 
of  scenarios.) 
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Third 

Orders 


j  Inadequate  Sc&ervtsian  j 


Supervisory  Conditions 


Fail  to  Provide  Guidance 
Fail  to  Provide  Oversight 
Fail  to  Provide  Training 
Fail  to  Track  Performance 
Fail  to  Track  Qualifications 
Failure  to  Inspect 


tin  appropriate  Operational 


Insufficient  Resources 
Inadequate  Brief  Times 
Improper  Manning 
Inadequate  Schedule 


Documents  not  updated 
No  Corrective  Actions 
[Unsafe  Condition  not  Reported] 


Ignoring  Risks 
Fail  to  Enforce  Rules/SOPs 
Use  of  Unsafe  Equipment 
Use  of  Untrained  Personnel 


And  under  the  Squadron  Second  Order  Supervisory  Condition,  we  have  these  Third 
Order  Error  Categories: 

Inadequate  Supervision  A  supervisor  who  does  not  ensure  that  maintenance  personnel 
are  wearing  required  personal  protective  gear 

Inappropriate  Operations  A  supervisor  who  directs  a  maintainer  to  perform  a  task 
without  considering  risks,  such  as  driving  a  truck  through  a  hangar 
Uncorrected  Problem  A  supervisor  who  neglects  to  correct  maintainers  who  routinely 
bend  the  rules  when  they  perform  a  common  task 

Supervisory  Misconduct  A  supervisor  who  willfully  orders  a  maintainer  to  wash  an 
aircraft  without  proper  safety  gear 
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Let  us  now  examine  the  Working  Conditions  First  Order  Error  Category  breakdown. 
(Again  advise  students  to  follow  along  with  their  copy  of  the  HFACS-ME  Framework  slide), 
(quickly  go  to  next  slide) 
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BEEWSSgUl 


Night  visibility 
Workspace  Illumination 
Inadequate  flashlights 


srfBggMIiTOIl 


Extreme  Temperatures 
Ice  on  Equipment 
Visibility  in  Rain/Snow/Fog 
Equipment/Manning  Changes 
Inadequate  Clothing 


Slips/Falls 

Frostbite 

Sunstroke 


Working  Conditions  have  Second  Order  Error  Categories  of  Environment, 
Equipment,  and  Workspace  factors. 


The  Environment  Second  Order  can  be  further  broken  down  into  the  following 
Third  Orders: 

Lighting/Light  A  maintainer  who  is  working  at  night  on  the  flight  line  does  not  see  a 
tool  he  left  behind 

Weather/Exposure  A  maintainer  who  is  securing  an  aircraft  in  a  driving  rain  fails  to 
properly  attach  the  chains 

Environmental  Hazards  A  maintainer  who  is  working  on  a  pitching  deck  falls  from  the 
aircraft. 


(Note:  Environmental  Factors  are  often  not  adequately  reported.) 
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1  Working  Conditions  | 

UK 

Third 

Orders 


\  Damaged  j 

I  Unavailable  i 

Limited  Use 

Unreliable 

Gauge/Calibration  Errors 
Unsafe  (brakes,  electrical) 

Equipment  used  elsewhere 
Equipment  not  in  inventory 
Equipment  unusable 

Calibration  Expired 

Open  Purchase/Uncertified 
Extended  Beyond  Service 'Life 

Equipment  Third  Orders  are: 

Damaged  A  maintainer  who  is  using  a  defective  test  set  does  not  pre-check  it  before 
troubleshooting 

Unavailable  A  maintainer  who  starts  working  on  landing  gear  without  a  jack  because  all 
are  being  used 

Dated/Uncertified  A  maintainer  who  uses  an  old  manual  because  a  CD-ROM  reader  is 
not  available. 

(Note:  Tools  and  Equipment  Factors  are  also  not  adequately  reported.) 
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And  finally.  Workspace  factors  can  be  separated  into  these  Third  Order  Error 
Categories: 

Confining  A  maintainer  who  is  working  in  a  hangar  bay  cannot  properly  position  the 
maintenance  stand 

Obstructed  A  maintainer  who  is  spotting  an  aircraft  with  his  view  obscured  by  catapult 
steam 

Inaccessible  A  maintainer  who  is  unable  to  perform  a  corrosion  inspection  that  is 
beyond  his  reach. 


(Note:  Workspace  factors  are  also  not  adequately  reported.) 


82 


Slide  21 


Maintainer  Conditions,  the  conditions  that  uniquely  affect  individual  maintainers,  are  separated  into 
Medical,  Crew  Coordination  and  Readiness  Categories,  (quickly  go  to  next  slide) 
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The  Medical  Third  Orders  are  based  on  an  individuals  mental  and  physical  abilities 
(and  limitations)  to  perform  a  task  at  a  certain  time. 


They  include: 

Mental  State  A  maintainer  who  has  a  marital  problem  and  cannot  focus  on  a 
maintenance  action 

Physical  State  A  maintainer  who  worked  for  20  hours  straight  and  suffers  from  fatigue 
Limitation  (Physical)  A  maintainer  who  is  short  cannot  visually  inspect  an  aircraft 
before  it  is  launched 


(Note:  Medical  Factors  are  not  reported  adequately) 
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Maintainer  Conditions 


oorcUnanon 


Third 

Orders 


t  Communication  1 

1  Assertiveness  1 

Terms  not  Standardized 
Hand  Signals  not  Standardized 
Documentation  Failures 
Documentation  Delays- 
Equipment  Failures  (radios) 
Equipment  use  (lights/whistles' 
Inadequate  Brief 

Fail  to  Brief/Make  Suggestions 
Fail  to  Correct  Discrepancies 
Fail  to  Confirm  Messages 
Inattention  to  Feedback 
Waiver  when  Confronted 

f Adaptabilitv/Ftexibilitv  ! 


Emergency  Response 
System  Failure  Response 
Changes  to  Routine 
Team  member  changes 


Crew  Coordination  factors  include  problems  in: 

Communication  A  maintainer  who  leads  a  taxiing  aircraft  into  another  due  to  improper 
hand  signals. 

Assertiveness  A  maintainer  who  performs  a  task,  not  in  accordance  with  standard 
procedures,  because  the  maintainer  was  overly  submissive  to  a  superior. 
Adaptability/Flexibility  A  maintainer  who  downplays  a  downing  discrepancy  to  meet 
the  flight  schedule. 
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Not  Trained  for  Task  1  Not  Certified  in  Task  1  Intoxicated  at  W  ork 
Not  Current  in  Training  I  Not  Certified  in  Model  I  Hung  over 

Unrealistic  Training  I  Qualifications  Expired  •  ■  v-  Drug  use 


ineffective  OJT 


And  the  last  Malntainer  Conditions  category  to  discuss  is  Readiness.  Its  Third 
Orders  involve: 

Training/Preparation  A  maintainer  who  is  working  on  an  aircraft  skipped  the  requisite 
OJT  evolution 

Certification/Qualification  A  maintainer  who  engages  in  a  procedure  that  he  or  she  has 
not  been  qualified  to  perform 

Infringement  A  maintainer  who  is  intoxicated  on  the  job. 

(Note:  Readiness  Factors  are  also  not  reported  adequately.) 
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Missed  Communication 
Loss  of  Situational  Awareness 
Distracted 
Fail  to  Secure 
Left  Unattended 
Improper  Lookout 


Procedure  Mistakes 
Missed  Task 
Sequence  Errors 


Insufficiently  Trained 
Insufficient  References 
Incorrect  Procedures 


Delayed  Response 
Overuse  of  Controls 
Omitted  Step  in  Procedure 
Omitted  Checklist  Item 
Poor  Technique 
Improper  Cross-Check 


Exceeded  Ability 
Poor  Decision 
Wrong  Response 
Misdiagnosed  Problem 
Improper  Procedure 


Maintainer  Errors  are  of  the  following  Third  Order  types: 

Attention  A  maintainer  who  misses  a  hand  signal  and  backs  a  forklift  into  an  aircraft 
Memory  A  maintainer  who  is  very  familiar  with  a  procedure  may  reverse  steps  in  a 


sequence 

Knowledge/Rule  Based  A  maintainer  who  inflates  an  aircraft  tire  to  a  pressure  required 
by  a  different  aircraft 

Skill  Based  A  maintainer  who  roughly  handles  a  delicate  engine  valve  will  cause  undue 
damage 

Judgment/Decision-Making  A  maintainer  who  fails  to  make  appropriate  decisions  due 
to  limited  information,  inadequate  preparation,  or  perceived  pressure  to  perform 
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Maintainer  Acts  ^ 

■HV 

_ 

Third 

Orders 


i  Routine  1 

Infraction  1 

Did  Not  Follow  Brief 
Bending  of  Regulations/SOPs 
Use  of  Incorrect  Equipment 
Violated  Training  Rules 

Do  Not  Utilize  Checklists 

jgqpj 

Single  Event  to  Save  Time 
Violation  to  Expedite  Mission 
Use  of  Incorrect  Equipment 
Skip  Publication  Crosscheck 

~'T 

\ :  Flagrant  1 

j  Sabotage  1 

Unauthorized  Procedure 

Not  Current/Qualified 
Exceed  Equipment  Limits 
Thrill  Seeking 

mm 

Deliberate  destruction 
Attempt  to  injure/kill 
Purposeful  Mission  Disruption 

There  are  also  several  types  of  Maintainer  Violations.  Their  Third  Order  category 
is  based  upon  frequency,  intent,  and  supervisory  involvement. 

Routine  A  maintainer  who  engages  in  practices,  condoned  by  management,  that  bend 
the  rules 

Infraction  A  maintainer  who  strays  from  accepted  procedures  to  save  time,  bending  a 
rule 

Flagrant  A  maintainer  who  willfully  breaks  standing  rules  disregarding  the 
consequences 

Sabotage  A  maintainer  who  performs  a  criminal  act  to  destroy  equipment  or  endanger 
the  lives  of  others 
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(FY  90-97)  Human  Error  in 
Maintenance  Related  Mishaps 


470  Maintenance  Related 
Mishaps  Were  Analyzed  For 
Human  Errors 

Classification  Process 
Performed  by  Naval 
Maintenance  Personnel 


HFACS,  as  previously  stated,  has  been  used  effectively  within  the  Navy,  Without  reopemn 
investigation,  HFACS  was  applied  to  existing  Mishap  Investigation  Reports, 

(go  to  next  slide  for  general  findings). 
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General  Findings 

>  Poor/non-existent  maintenance 
procedures 

>  Inadequate/poor  supervision  of 
maintenance  evolutions 

>Miscommunication  -  supervisor 
to  subordinate,  pass-down,  or 
shift  turnover 

>Not  using,  lack  of,  or  outdated 
publications 

>  Violations  -  not  following  policy, 
procedures,  checklists,  etc. 


This  slide  shows  the  general  findings  of  the  analysis.  I’m  sure  we  are  all  familiar  with  these  latent 
conditions.  Let  us  now  look  at  another  study  of  63  Class  A  mishaps  between  FYs  90-97.  (go  to  next  slide) 
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HFACS-ME  Results  of  MRMs 


First  Order  Analysis  of  65  Maintenance  Related  Mishaps 


HFACS  ANALYSIS  OF  Class  a  FYs  90-97  Maintenance  Related  Mishaps  (MRMs). 

During  FYs  90-97  there  was  a  total  of  63  Class  A  Mishaps,  of  which  61  were  Flight,  0  were  Flight  Related,  and  2  were 
Aircraft  Ground.  Two  Navy  Maintenance  Officers  and  two  Navy  Chiefs  used  the  HFACS  Maintenance  Extension  to 
classify  the  human  factors  causes  reported  in  these  mishaps.  They  discovered  the  following  profile  of  human  errors: 

Management  Conditions  -  67%  of  all  Naval  Aviation  Class  A  MRMs  reported  Squadron  Supervisory 
Conditions,  whereas  21%  had  Unforeseen  Supervisory  Conditions  (not  shown). 

Maintainer  Conditions  -  21  %  of  all  Naval  Class  A  MRMs  reported  Medical,  CRM,  or  Readiness  Maintainer 
Conditions.  Note:  Maintainer  Conditions  were  under  reported ,  more  are  likely  present  and  have  an  effect 
Working  Conditions  -  3%  of  all  Naval  Class  A  MRMs  reported  Environment  Equipment  or  Workspace 
Working  Conditions.  Note:  Workspace  Conditions  were  under  reported,  more  are  likely  present  and  have  an  effect 
Maintainer  Acts  -  75%  of  all  Naval  Aviation  Class  A  MRMs  reported  Maintainer  Errors,  whereas  40%  had 
Maintainer  Violations. 

Clearly,  latent  conditions  in  the  form  of  Supervisory,  Maintainer.  and  Workspace  factors  are  present  that  can  impact 
maintainors  in  the  performance  of  their  jobs.  However,  many  Maintainer  and  Workspace  Conditions  are  not  reported 
due  to  the  reporting  system  in  place,  perceptions  of  accident  causation,  or  culture/climate  issues.  Specifically, 
inadequate  supervision  of  maintenance  evolutions,  not  ensuring  personnel  are  trained  and/or  qualified,  not  enforcing 
rules,  and  poor  communication  characterize  the  majority  of  latent  Supervisory'  Conditions.  Poor  pass  down, 
coordination,  and  communication;  non-use  or  lack  of  publications,  policies,  and  procedures;  and  fatigue  comprise  most 
latent  Maintainer  Conditions.  Finally,  most  Maintainer  Errors  reflect  a  lack  of  training,  experience,  and  skill,  whereas 
Maintainer  Violations  consist  of  routine  non-compliance  with  standard  procedures  and  practices,  and  infractions, 
bending  the  rules  in  order  to  meet  mission  requirements  and  the  flight  schedule. 

Conclusions 

HFACS-ME  was  effective  in  capturing  the  nature  of  and  relationships  among  latent  conditions  and 
active  failures  present  in  Class  A  MRMs.  The  insights  gained  provide  a  solid  perspective  for  the 
development  of  potential  intervention  strategies.  The  major  mishaps  analyzed  were  primarily  FMs,  meaning 
that  many  imposed  in-flight  Maintenance  Conditions  on  aircrew.  During  FYs  90-97  there  were  almost  500 
MRMs  in  Naval  Aviation,  many  of  which  were  of  lesser  severity  and  were  either  Flight  Related  or  Aircraft 
Ground  Mishaps.  Such  mishaps  involve  primarily  ground  and  ramp  activities  and  can  lead  directly  to  a 
mishap  or  injury.  Consequently,  the  present  profile  and  observed  relationships  only  hold  for  the  mishaps 
considered  and  cannot  be  generalized  to  all  MRMs.  Further,  it  can  be  contended  interventions  developed  for 
major  mishaps  that  primarily  involving  maintenance  activities  such  as  engine  repair  are  likely  not  appropriate 
for  ones  of  lesser  severity  that  involve  other  activities  such  as  loading  ordnance  or  towing  aircraft. 

Presently,  an  in-depth  analysis  of  all  MRMs  is  underway  and  it  is  planned  to  contrast  major  vs.  minor 
MRMs,  MRMs  occurring  during  maintenance,  stores,  and  ramp  activities,  and  MRMs  for  specific 
communities. 
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HFACS-ME  Summary 


>  HFACS-ME  is  effective  in  classifying 
aviation  maintenance  causal  factors 

>  HFACS-ME  enables  organizations  to 
develop  successful  accident  intervention 
strategies 

>  HFACS-ME  should  be  applied  to  both 
major  and  minor  accidents  to  fully  analyze 
maintenance  errors 


Presentation  Summary: 

The  HFACS-ME  model,  as  you  have  just  seen,  has  demonstrated  effectiveness. 

With  HFACS-ME,  a  safety  organization  can: 

-  effectively  classify  (code)  maintenance  related  accident  causal  factors 

-  develop  specific  intervention  strategies  to  mitigate  causal  factors 

NOTE:  Because  most  major  mishaps  occur  during  flight  operations,  it  is  essential  to  also  evaluate  the 
“minor”  mishaps  and  incidents  that  occur  on  the  ramp  and  in  the  hangar.  Such  mishaps  involve  activities 
that  can  lead  directly  to  damage  to  the  aircraft  or  injury  to  the  maintainer. 
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Maintenance  Error  Information 
Management  System  (MEIMS) 


■  H  FACS- ME 

■  Taxonomy 

■  Database  Tool 

■  Prototype  developed 

■  at  NPS 


■  Second  Revision  (Nov  2000) 


Now  that  we  have  some  understanding  of  the  HFACS-ME,  lets  take  a  look  a  the 
prototype  database  tool  which  uses  that  taxonomy-  It  is  called  MEIMS, 
Maintenance  Error  Information  Management  System. 


This  is  an  Access  2000  based  prototype,  developed  at  NPS,  the  latest  revision  just 
recently  completed. 


94 


Slide  33 


The  Main  Menu 

Notice  the  five  button  functionality,  plus  exit  Placing  your  pointer  over  any 
of  the  menu  selections  will  give  you  a  brief  description  of  that  selection. 
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sagggj 


Iircraft  Model 

tk>  Fl|»4r. 


SefeatwooCTore. 
CH'f 'ior*  Trom  theie: 


Branch  of  Service 

USN  USMC 

Location  of  Mishap  J 

PflJ-  2  ii\ 


mWiM: 

mmmm 


Returnt©  Maity  Menu 


The  query  menu. 

The  user  may  select  one  or  many  (using  the  Multiple  Criteria  button)  choices 
from  the  left  and  get  mishap  information  regarding  that  particular  selection(s). 
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Maintenance  llshapQneiy 

Mshaptote - .  ]T  ’  .  : 


QassofMishap 


11/28/139 


Branch  d  Sevic 


Acft  on  FCF  had  teft  eng  cJ  ho(  Ight  during  dsr; 


Workdrsw?;.  andCOlfafedtopro'^eadequatetechd^aandprocedtiesduelocaiffecer  SC  SQN  JOQ 


This  is  the  TACAIR  selection.  Notice  that  the  selected  criteria  is  in  blue.  There  is  a 
brief  description,  contributing  factors  and  the  codes  for  Levels  1, 2  &  3.  Definitions 
for  those  codes  can  be  found  by  clicking  on  the  “Define  HFACS  Codes”  button. 
Also,  at  the  bottom,  the  number  of  mishaps  meeting  that  criteria. 
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Bndf  Desc^ion 
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Arcraft  Category  .  jHeb 
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D3rti>ing  Factors 
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■  Leveil .  level  2  level  3 


MCOFaiedte  Provide  $7  SystotoDocaaert^SttoihADB 
IKADQ  Dedgn  d  Sonar  Seat  Sv^chAlows  Safe  Water  l^juson/Cortock^ 


SC  SQK  -IDS 
SC  :  ORG  I DES 


<Sad 


Record:  Ml 


1  >  l»» Wof  i9:®ere»d5 


...with  the  selection  criteria  in  blue.  Define  HFACS  Codes  allows  the  user  to  view 
the  codes  for  each  level... 
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Supervisory  *■■■■■■■■■ 
Conditions  (SC)  (SOK) 


Medici  (MH)> 
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Working  Equipment  (EOP)  I  Damaged  (D&S)  ftJNA)  •  Dated.5* 

Conc^tkms  (WC)  |  UeertiWflXIO 
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Error  (ERR) 
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...traveling  right  to  left  in  Levels  1, 2  &  3 
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Select  Mishap  Data  By: 


Aircraft  Model 

F-RH-4S... 

Aircraft  Type 

'  TACAJRHeto... 

Branch  of  Service 

USN.USMC  .  t; 

Location  of  Mishap 

v-  3  Ashore^Emharked., 

Mishap  Class 

A  8,  C 

Mishap  Type 

m,Fm,  aqu 

Year  of  Mishap 

■-•-.-Cslenda r 


; _ 1  Multiple  Criteria 

Seieettsro  or  more 
|;3  criterion  from  the  left 


■  !  :  ■  '  ■  /  : 


HFACS-ME  Summary 

fey  ME  Category  using  s 
Muifiple  Often®  Query  ,  - 

Return  to  Main  Menu 


'  |p; 


And  Selecting  HFACS-ME  Summary  gets  the  user... 
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...This  presentation.  Notice  that  the  user  can  modify  the  choices  directly  from  this 
screen  (F-14/Class  B  selected).  This  summary  page  is  good  for  pulling  out  data 
specific  to  levels  1-3  or  within  levels. 
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Returning  to  the  Main  Menu,  we  will  now  look  at  the  Graphing  Function 
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Expert  Graph 


Expert 
Graph  Menu 


1  .Select  X-Axis  and  Y-Axis 
categories  by  checking  the 
option  button  to  the  left  of 
the  category 

2.  Select  specific 
values  for  X  and  YAxes  by 
decking  on  the  "Select  Axis 
Values’*  command  button 
at  the 
bottom. 

3.  Dick  on  the  “Graph 
lt“  command  button  to 
create  the  graph  (vsi  not 
be  available  un$  both  axis 


XAxis  YAxis  ^ 


:  C  1  Aircraft  Model  i 

!  -:'-r  :  ;  Aircraft  Type  :  ^  ; 


r  r 
r  l  r  \ 


C  \  C  [  Mish&pCiass. 

“  '  “  ~  '  *  **  *  tf* 


v  ::sr 


C  ;  r  Type  of  Mishap 

r  r  HFACS-ME  Level  One 

;  r  :  r  HFACS-ME  Level  Two 


«...  '■  i 


CH  r  HFACS-ME  Level  Th;ee 


Sdect  X  Axis  Values  SelectYAxfe  Values^  j 

■  .  :  : . ;■ /  :  ■ 

<  Back  | 


The  Graphing  Function  allows  the  user  to  choose  which  criteria  will  be  on  which 
axis,  and  then... 
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Both  the  X  and  Y  axis  may  be  further  defined  to  produce 
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Selecting  the  REPORT  MENU  gets 
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a  choice  of  multiple  reports 
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A!!  Maintenance  Related  Mishaps  by  Branch  of 
Service  (1  389-1999) 
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...in  this  case,  the  Mishap  Distribution  by  Service. 
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Enter  New  Maintenance  Mishap  Date  AddNewRecotd 


Close  Forni 
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The  Investigation  Tool  is  not  yet  fully  functional,  but  will  allow  the  user  to  “drill 
down”  into  the  specifics  an  individual  mishap  to  better  ascertain  learning  point. 

Last,  but  not  least.  Exit  This  button  exits  the  MEIMS  program  and  Microsoft 
Access  2000. 


112 


Slide  51 


This  concludes  the  Tutorial.  You  now  have  a  better  understanding  of  the  individual 
error  categories  and  their  interrelationships  in  a  mishap’s  chain  of  events. 
Additionally,  you  have  seen  some  of  the  functionality  of  the  MEIMS  tool.  If  there 
are  no  questions,  please  follow  the  instructor’s  directions  regarding  the  usability 
test. 
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APPENDIX  C 


MAINTENANCE  ERROR  INFORMATION  MANAGEMENT  SYSTEM  (MEIMS) 

EVALUATION 

Background.  Thank  you  for  participating  in  a  usability  study  (evaluation)  of  a 
prototype  tool  for  the  Maintenance  Error  Information  Management  System  (MEIMS). 
This  tool  was  developed  at  NPS  and  has  been  modified  based  upon  previous  usability 
studies.  This  study  is  being  conducted  by  LCDR  Mike  McCracken,  USN  as  part  of  a 
thesis  project  for  his  Master  of  Science  program  in  Information  Technology 
Management.  MEIMS  was  developed  to  address  and  identify  patterns  of  human  error  in 
Naval  Aviation  maintenance-related  aircraft  mishaps.  The  Human  Factors  Analysis  and 
Classification  System  Maintenance  Extension  (HFACS-ME)  taxonomy  is  the  foundation 
of  MEIMS  and  is  an  effective  method  for  classifying  and  analyzing  the  presence  of 
human  error  in  maintenance  operations  leading  to  major  mishaps,  accidents  of  lesser 
severity,  incidents  and  maintenance  related  personal  injury  cases.  A  previous  study 
indicated  that  prior  to  using  MEIMS  a  tutorial,  explaining  the  taxonomy  and  some 
features,  would  be  beneficial.  This  study  compares  two  groups  of  users,  one  which 
receives  the  tutorial  and  one  that  does  not.  Feedback  from  user  groups  will  be  used  to 
further  improve  MEIMS  and  the  tutorial. 

MEIMS  captures  maintenance  error  data,  facilitates  the  identification  of  common 
maintenance  errors  and  associated  trends,  and  supports  understanding  of  how  to  identify 
human  errors  in  the  future.  The  target  audience  for  this  information  management  system 
tool  includes  safety  personnel  (data  entry  &  retrieval  by  unit  safety  officers,  other  safety 
&  training  personnel,  maintenance  officers,  maintenance  supervisors),  mishap 
investigators-for  data  retrieval  (Aircraft  Mishap  Board  members,  squadron  safety 
officers),  and  analysts  (from  the  Naval  Safety  Center,  the  command’s  safety  officer  or 
one  from  its  higher  headquarters).  This  tool  allows  can  directly  lead  to  a  decreased 
mishap  rate  and  overall  increased  mission  readiness  due  to  the  training  and  analysis 
opportunity  it  provides. 

Usability  Study.  You  will  be  given  a  packet  of  instructions  to  guide  you  through 
MEIMS.  You  will  be  asked  to  make  comments  on  the  effectiveness  and  usability  of  the 
prototype  system  during  your  testing  phase.  Additionally,  you  will  be  asked  to  complete 
an  “exit  survey”  after  completion  of  your  testing.  Questions  will  include  demographic 
information,  objective  questions  about  MEIMS  usability,  questions  regarding  the  tutorial 
and  its  effectiveness  (if  applicable),  and  subjective  questions  and  comments  for  areas  not 
covered  in  the  objective  section.  The  study  should  take  no  more  than  15-20  minutes. 

Completion  of  Study.  Upon  completion  of  your  testing  and  survey  you  will  be 
asked  to  return  your  packet  of  instructions  to  LCDR  McCracken.  Thank  you  again  for 
your  participation. 


Mike  McCracken 
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Instructions  for  Maintenance  Error  Information  Management  System  (MEIMS) 

Tool  Evaluation 


Start-up 

1.  Go  to  a  computer  lab  with  OFFICE  2000  loaded.  Log  on  into  the  NPGS  domain. 
Open  Microsoft  ACCESS  2000,  then  using  the  zip  disk  provided,  open  the  file  Hfac2000. 

Question  1:  Did  you  have  any  problems  accessing  the  program?  Y/N  (circle  one ) 

If  so,  please  describe: 


Main  Menu 

2.  You  will  now  have  the  Main  Menu  displayed  with  the  world  famous  Supersonic 
Hornet  photo  in  the  background. 

3.  Note  the  six  categories  next  to  the  command  buttons  on  the  bottom  right  portion  of  the 
screen.  The  system  has  “focus”  on  “Query  Menu”.  Note  the  information  on  this  button  in 
the  bottom  left  gray  buffer  above  the  Windows  Start  button.  Place  the  mouse  pointer 
over  the  Query  Menu  box  (don’t  click,  if  you  do,  select  <Back>  on  subsequent  page)  and 
note  information  that  appears  in  the  Text  Box  (both  of  these  sources  of  information  will 
be  available  throughout  MEIMS). 

4.  Select  <Tab>  and  view  the  same  information  for  the  remaining  four  command  buttons 
(note,  if  you  select  <Exit>  you  will  have  to  re-enter  the  system  (see  step  1  above). 

Question  2:  Is  the  terminology  clear  enough  to  understand  what  each  of  the  four 
command  buttons  does?  If  not,  what  could  be  changed  to  make  it  clearer? 


5.  Select  (click  or  tab  to  &  enter)  <Query  Menu> 


Query  Menu 

6.  Note  there  are  two  sections  on  the  Query  Menu.  The  left  half  of  the  screen  has  seven 
categories  to  help  you  define  how  you  would  like  to  view  the  mishap  data.  The  right  half 
of  the  screen  has  four  command  buttons. 

7.  Select  <Aircraft  Model> 

8.  Another  form  appears:  “Query  by  Aircraft  Type”.  Select  your  type  aircraft,  then 
select  <View  Selectionx  “Summary  of  Mishap”  Form  appears.  Note,  your  aircraft 
selection  has  a  blue  background.  Review  the  “Brief  Description”  of  the  mishap  and  the 
“Contributing  Factors.”  View 
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Question  3:  What  aircraft  did  you  select? _ 

How  many  separate  mishaps  of  that  aircraft  type  are  in  the  database  ?, 
View  one  of  the  mishaps. 

What  are  the  level  3  codes  &  what  do  they  mean? _ 

How  did  you  find  that  info? _ 

When  you  are  through  viewing  the  data,  select  <Close  Form> 

Select  another  aircraft  type,  and  view  some  of  the  mishaps  (optional). 
When  complete  select  <Back>  on  Query  by  Aircraft  Model  Form 

9.  Select  another  category  (your  option)  &  view  the  data. 

Question  4:  Which  (if  any)  of  the  seven  categories  do  you  find  useful? 


Which  (if  any)  of  the  seven  categories  do  you  not  find  useful? 


10.  Select  <Multiple  Criteriax  Create  your  own  query  using  two  or  more  criteria. 
Question  5:  What  Criteria  did  you  choose? 


How  many  mishaps  for  your  selection  were  in  the  database? 
Did  you  find  this  function  useful?  Why  or  why  not? 


11.  Return  to  Query  Menu.  Select  <HFACS-ME  Summaryx 

Question  6:  How  many  total  mishaps  are  in  the  database? _ 

How  many  mishaps  have  a  level  one  category  of  Worker  Conditions  ?_ 
How  many  mishaps  have  a  level  two  category  of  Medical? _ 
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Further  define  the  system  by  your  aircraft  model  (or  select  another  type). 

Question  7:  What  aircraft  did  you  select? _ 

How  many  separate  mishaps  of  that  type  aircraft  are  in  the  database? _ 

Conduct  further  queries  as  desired.  When  complete,  return  to  Query  Menu  &  return  to 
Main  Menu. 

11.  Select  <Report  Menu>.  Review  the  six  command  buttons  and  their  functions. 

12.  Select  <Mishap  Distribution  -  Locations  Review  the  report  data.Find  your  type 
aircraft  (or  review  another)  in  the  report.  <Close>  the  report  &  return  to  the  Report 
Menu. 

Question  8:  How  many  Detached  mishaps  are  in  the  database? _ 

13.  Select  <A11  Mishaps-Chronological  Listing>.  Review  data.  <Close>  the  report  when 
complete  &  return  to  Report  Menu.  Return  to  Main  Menu. 

14.  Select  <Expert  Graph  Menu>.  Follow  directions.  Create  one  graph  with  aircraft 
model  (yours  and  1  or  2  others)  on  the  X-Axis  and  HFACS-ME  Level  One  (all  four 
codes)  on  the  Y-Axis. 

Question  9:  What  aircraft  did  you  select? _ 

Did  you  notice  a  difference  in  the  level  one  codes  between  the  aircraft  (if  so,  what)? 

Return  to  <Expert  Graph  Menu>.  Try  more  graphs  as  desired.  When  complete,  return  to 
Main  Menu. 

15.  Select  <Add  New  Data>.  Enter  two  mishaps  to  the  database: 

Question  10:  What  are  the  Mishap  Numbers  for  the  data  you  are  entering? 

Check  to  see  if  your  entries  were  added  to  the  database  by  Looking  at  the  end  of  the 
Chronological  Listing  on  the  Report  Menu  (look  for  your  Mishap  Numbers). 

Question  11:  Did  you  see  your  data  in  the  Chronological  Listing? _ 


16.  Return  to  Main  Menu  &  Select  Investigation  tool.  This  selection  is  not  yet  fully 
functional.  It  will  be  used  to  “drill  down”  a  specific  mishap  or  types  of  mishap  to 
determine  if  like  factors  occur.  It  will  provide  enhanced  analysis  and  training  benefit. 
You  may  choose  multiple  criteria,  just  as  in  query,  to  limit  narrow  your  focus  (optional). 

17.  Return  to  Main  Menu  &  Exit  the  Program. 

18.  Please  fill  out  the  Exit  Survey  Questionnaire. 
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APPENDIX  D 


MAINTENANCE  ERROR  INFORMATION  MANAGEMENT  SYSTEM  (MEIMS) 

EXIT  SURVEY 

User’s  Impression  of  the  Maintenance  Error  Information  Management  System  (MEIMS) 

Prototype  Tool 

Purpose:  This  survey  evaluates  a  user’s  overall  satisfaction  of  the  Maintenance  Error 
Information  Management  System  (MEIMS)  prototype  tool.  It  consists  of  four  parts. 

Part  I:  Demographic  Information.  Part  I  provides  the  user’s  aviation 
background,  computer  experience,  and  availability  of  software  and  hardware  systems 
used  in  the  Navy  and  Marine  Corps. 

Part  II:  User  Satisfaction  with  the  MEIMS  Prototype  Tool.  Part  II  deals  directly 
with  user  feedback  as  they  use  the  prototype  tool. 

Part  III:  User  Overall  Satisfaction  with  the  MEIMS  Prototype  Tool.  Part  III 
allows  users  to  give  general  feedback  about  the  prototype  tool. 

Part  IV:  (if  applicable)  User  Satisfaction  with  the  Tutorial.  Part  IV  allows  user 
to  provide  feedback  about  the  Tutorial 


Part  I.  Demographic  Information 

Follow  the  instructions  after  each  numbered  question  or  statement. 

1.  I  am  attached  to  a  command  that  primarily  performs  maintenance  (military  and/or 
civilian)  at  the: 

(Select  one  from  the  list  and  check  the  box) 

□  Squadron  Level 

□  Intermediate  Level  (AIMD) 

□  Depot  Level  (NADEP) 

□  Command  does  not  perform  aircraft  maintenance 

□  Other  (describe  if  other) _ _ 

2.  How  long  have  you  been  using  a  computer? 

(Select  one  from  the  list  and  check  the  box) 

□  Less  than  one  month 

□  One  month  to  less  than  one  year 

□  One  year  to  less  than  two  years 

□  Two  years  or  more 
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3.  What  software  do  you  normally  use? 

(Check  all  boxes  that  apply) 

□  Microsoft  Office  (Word,  PowerPoint,  Excel,  Access) 
What  version? 

(Check  all  boxes  that  apply) 

□  97 

□  2000 

□  not  sure  of  version 

□  other  (describe  if  other) _ 

□  Lotus  Smart  Suite  (Word  Pro,  Lotus  123...) 

What  version? 

(Check  all  boxes  that  apply) 

□  97 

□  9.5 

□  not  sure  of  version 

□  other  (describe  if  other) _ 

□  Corel  Word  Perfect  Office  (Word  Perfect,  Quattro  Pro...) 
What  version? 

(Check  all  boxes  that  apply) 

□  Corel  Office  7 

□  2000 

□  not  sure  of  version 

□  other  (describe  if  other) _ 

□  Other  (describe  if  other) _ 

4.  What  software  application  categories  are  you  familiar  with? 

(Check  all  boxes  that  apply) 

□  Word  Processing  (MS  Word,  Word  Perfect,  Word  Pro...) 

□  Spreadsheet  (Excel,  Lotus  123,  Quattro  Pro...) 

□  Presentations  (PowerPoint,  Harvard  Graphics...) 

□  Graphic  Software  (Corel  Draw,  Adobe  Photoshop...) 

□  E-Mail  (Outlook,  Eudora,  AOL...) 

□  Database  (Access,  DBase...) 
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5.  What  computer  operating  systems  do  you  use? 
(Check  all  boxes  that  apply) 

□  Windows  (3.1,  95,  98,  2000) 

□  Windows  NT 

□  Macintosh 

□  UNIX 

□  Linux 

□  Other  (describe  if  other) _ 


Part  II.  User  Satisfaction  with  the  Four  Sections  of  the  MEIMS  Prototype  Tool 

Select  the  category  that  best  matches  your  impression  of  each  of  the  below  categories 
(and  check  the  box). 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


I  feel  the  information 
on  the  MEIMS  tool  was 
in  a  logical  form 


I  found  the  MEIMS 
tool  easy  to  navigate 


My  tour  of  the  MEIMS 
tool  was  very  interesting 


The  information  presented  on 
the  MEIMS  tool  is  relevant  to 
maintenance  operations 


The  concept  of  the  MEIMS 
tool  is  a  good  one. 
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Part  III.  User  Overall  Satisfaction  with  the  MEIMS  Prototype  Tool 

Please  make  any  comments  on  the  MEIMS  Prototype  Tool  not  reflected  in  your 
comments  in  sections  1  and  2. 

The  most  positive  aspects  of  the  MEIMS  prototype  tool  were: 


The  most  negative  aspects  of  the  MEIMS  prototype  tool  were: 


I  would  make  these  changes  (if  any)  to  the  MEIMS  prototype  tool: 


If  you  received  the  Tutorial,  please  continue  on  to  Part  IV .  If  you  did  not  receive 
the  Tutorial,  you  are  complete.  Thank  you!  Your  participation  is  greatly 

appreciated! 
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Part  IV.  User  Satisfaction  with  the  Tutorial 

Select  the  category  that  best  matches  your  impression  of  each  of  the  below  categories 
(and  check  the  box). 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


I  feel  the  information 
in  the  Tutorial  was 

beneficial 


The  Tutorial  made 
the  evaluation  easier 
to  complete 


The  Tutorial  was  interesting 


The  information  presented  in 
the  Tutorial  is  relevant  to 
the  MEIMS  tool 


The  concept  of  a  tutorial 
is  a  good  one. 


Please  make  any  additional  comments  on  the  Tutorial. 
The  most  positive  aspects  of  the  Tutorial  were: 
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The  most  negative  aspects  of  the  Tutorial  were: 


I  would  make  these  changes  (if  any)  to  the  Tutorial. 


Thank  you!  Your  participation  is  greatly  appreciated! 
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